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CHAPTER  I 


INTRODUCTION 

Acquisition  of  reliable  avionics  equipment  with  lower 
operational  and  support  cost  is  crucial  if  we  are  to  meet  the 
growing  needs  of  increasingly  sophisticated  avionics  systems  and 
subsystems  required  to  accomplish  present  day  and  future  air  mis- 
sions. The  existing  Department  of  Defense  managerial  approaches 
are  not  adequate  to  meet  this  challenge.  A few  examples  are  included 
in  this  report  to  demonstrate  the  foundation  of  this  view. 

Today  each  of  the  services  is  faced  with  less-than-desired 
combat  effectiveness  levels  and  rising  life  cycle  cost  for  new 
weapons  systems.  This  situation  exists  because  reliability  require- 
ments have  not  been  adequately  specified  or  enforced.  The  lack  of 
adequately  reliable  avionics  contributes  to  the  cost  of  ownership 
in  many  ways;  cost  of  down  time,  cost  of  failure,  pilot  safety,  the 
direct  cost  of  maintenance  spares,  labor,  training,  personnel,  etc. 

As  an  example,  the  Navy  is  currently  spending  $0.5  billion  per  year 
in  a peace-time  environment  for  spare  parts  alone,  to  repair  some 
two  million  failures."^  It  is  the  opinion  of  this  writer  that  the 
reliability  achieved  for  a military  system  is  directly  dependent 
upon  the  reliability  requirements  imposed,  and  upon  the  management 
emphasis  placed  on  reliability  by  both  the  government  and  the 
contractor  throughout  the  life  cycle  of  the  equipment. 

^Admiral  Isaac  C.  Kidd  made  this  statement  in  describing  the 
supply  and  depot-level  maintenance  pipeline  (Defense  Management 
Journal , April  1976,  p.  13). 
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statement  of  the  problem 

The  problem  addressed  is  the  Engineering  Development  (Test- 
Analyze-Find  and  Correct)  testing  of  avionics  systems  and  subsystems 
and  the  impact  of  implementing  the  C version  of  MIL-STD  781. 

The  increased  complexity  of  today's  avionics  subsystems,  the 
associated  maintenance  and  operational  support  cost  coupled  with 
the  less-tuan-desired  level  of  reliability  is  the  subject  of  wide 
interest  throughout  the  Department  of  Defense. 

One  of  the  areas  that  has  nationwide  interest  is  the  lack  of 
correlation  between  the  demonstrated  reliability  test  data  in  the 
laboratory  and  the  observed  reliability  test  results.  The  problems 
associated  with  the  subject  engineering  development  testing  of 
avionics  equipment  is  considered  a significant  problem  by  both 
government  and  industry. 

The  growing  awareness  and  concern  by  the  Congress  of  the 
relationship  between  life  cycle  cost  and  unreliable  equipment  are 
reflected  in  the  criticism  by  Congress  of  our  current  acquisition 
policies.  This  condition  plus  the  limited  funds  available  to 
acquire  new  weapons  requires  industry  and  the  government  to  work 
more  closely  and  to  draw  upon  the  knowledge  of  each  other  to  resolve 
this  situation. 

With  this  in  mind,  the  many  factors  which  influence  reliability 
of  avionic  subsystems  will  be  discussed. 


2 


PURPOSE  OF  THE  STUDY 


The  purpose  of  this  study  is  to  explore  the  reliability 
problems  associated  with  engineering  development  (Test-Analyze- 
Find  and  Correct  Phase)  testing  of  avionics  subsystems  and  equip- 
ment; to  examine  some  of  the  implications  of  the  C version  of  MIL 
Standard  781;  to  identify  some  of  the  significant  issues  and  prob- 
lems associated  with  achieving  reliability  for  avionic  subsystems; 
and  to  attempt  to  provide  recommendations  that  may  result  in 
improved  reliability  and  contribute  to  reduction  in  the  cost  of 
ownership . 


BACKGROUND  INFORMATION 


For  years  we  have  observed  significant  variances  between  the 
reliability  of  avionics  equipment  in  the  field  under  operational 
conditions  and  that  demonstrated  in  the  laboratory.  The  specific 
reasons  for  such  variance  have  been  the  subject  of  many  studies 
sponsored  by  both  the  DOD  and  industry. 

Despite  management  efforts  and  technological  improvements  made 
during  the  past  decade,  the  reliability  of  avionics  systems  and 
subsystems  is  still  lower  than  desired.  Because  low  reliability 
ultimately  degrades  military  readiness  and  increases  cost  of  owner- 
ship, all  of  us  associated  with  the  military  can  appreciate  the 
ramifications  of  this  situation. 

Efforts  are  now  under  way  in  the  Department  of  Defense  to  improve 
this  condition.  In  the  future  greater  emphasis  will  be  placed  on  both 

reliability  design  and  realistic  testing  during  the  early  phases  of 
the  acquisition  process. 
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DOD  and  industry  are  jointly  working  toward  increasing  the 
realism  of  laboratory  testing.  The  basic  concept  is  to  combine 


performance,  reliability  and  environmental  testing  (combined  stress) 
insofar  as  practical.  Colonel  Ben  Sweet,  ODDR&E,  identified  this 
subject  as  one  of  the  missing  links  in  the  present  MIL-SPF.C-MIL-STD 
arrangement  between  DOD  and  industry.  This  new  approach  is  the 
basis  of  this  study. 

SIGN IF ICANCE  OF  THE  STUDY 

The  significance  of  the  study  is  derived  from  its  potential 
contributions  to  the  understanding  of  the  problems  associated  with 
avionics  reliability  and  the  possible  development  of  a system  that 
would  permit  the  continuous  assessment  of  reliability  during  the 
equipment's  entire  life  cycle.  This  would  result  in  increased 
combat  readiness  and  lower  logistic  cost;  The  results  of  this  study 
will  be  forwarded  to  DOD,  Office  of  the  Director,  Defense  Research 
and  Engineering.  The  recommendations  and  findings  of  this  study  are 
timely  as  an  input  to  the  new  DOD  Directive  on  Reliability  and  Main- 
tainability of  System  and  Equipment  (5000. X).  This  draft  directive 
is  currently  being  circulated  for  comments  and  inputs. 

QUESTIONS  TO  BE  ANSWERED 

For  the  purposes  of  this  study,  a set  of  objectives,  in  the 
form  of  questions  to  be  answered,  was  developed.  These  questions 
provided  guidelines  for  the  literature  research  and  for  the  personal 
interviews : 

A 


1.  What  is  your  perception  of  the  current  DOD  reliability  test 
phi losophy? 

2.  Do  you  think  DOD  has  a common  definition  of  what  constitutes 
a reliable  system,  i.e.,  field  reliability  to  the  logistician  may  not 
mean  the  same  thing  to  the  designer  or  contract  administrator? 

3.  Does  the  need  exist  for  DOD  to  provide  more  guidance  or  a 
policy  directive  to  establish  a more  realistic  reliability  program? 

4.  What  factors,  other  than  environmental,  influence  poor 
reliability  in  the  field? 

5.  In  your  opinion,  do  we  adequately  monitor  those  factors 
that  effect  equipment  reliability? 

6.  Can  the  reliability  of  avionic  subsystems,  be  improved  by 
improving  the  reliability  test  methodology  during  the  early  phase 
of  the  acquisition  cycle? 

7.  Do  we  do  an  adequate  job,  contractually,  defining  reliabil- 
ity requirements  (for  avionic  subsystems)? 

8.  Does  the  current  logistic  reporting  system  provide  the 
kind  of  data  from  the  field  necessary  to  improve  the  reliability  of 
avionic  subsystems? 

9.  Does  the  DOD  Representative  provide  the  equipment  designer 
and  the  reliability  engineers  adequate  information  during  the  initial 
phase  of  the  program? 

10.  What  is  your  perception  of  our  ability  to  provide  the 
contractors  with  realistic  mission  profiles? 

11.  How  much  reliability  consideration  should  be  given  to  the 
conceptual  and  initial  design  phase? 
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12.  How  much  consideration  should  be  given  to  the  reliability 


requirements  during  engineering  development  testing? 

13.  What  should  be  the  role  of  the  Government  representatives 
during  the  engineering  development  test,  i.e.,  test  find  and  fix 
phase? 

14.  What  are  your  perceptions  of  the  cost  impact  of  imple- 
menting the  C version  of  MIL-STD  781? 

15.  If  we  implement  the  C version  of  MIL-STD  781,  will  we  have 
closer  correlations  between  lab  test  results  and  field  reliabilitv  data? 

LIMITATIONS  OF  THE  STUDY 


This  is  an  exploratory  study  of  avionics  subsystems  reliability. 
The  development  of  this  study  was  restricted  by  the  scheduled  time 
as  defined  by  the  US  Army  War  College  for  research  projects.  All 
major  data  collection  and  interview  sessions  were  completed  between 
17  March  and  28  April  1977.  These  activities  were  conducted  on  a 
noninterference  basis  with  other  school  requirements. 

The  agencies  and  personnel  selected  to  participate  in  the  study 
were  not  all  encompassing.  Key  representatives  of  various  agencies 
whose  primary  responsibilities  are  concerned  with  reliability  issues 
were  asked  to  participate. 

The  results  of  the  study  were  based  on  the  analysis  of  data 
obtained  from  personal  interviews  and  a review  of  related  literature. 

ASSUMPTIONS  OF  THE  STUDY 

The  following  assumptions  were  made  prior  to  initiating  this 


study . 
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1.  The  current  emphasis  by  DOD  to  improve  reliability  of 


avionics  equipment  may  lead  to  new  managerial  strategies  that  will 
reduce  the  cost  of  ownership. 

2.  Reassessment  of  current  methodologies  used  in  conducting 
reliability  testing  and  establishing  reliability  requirements  may 
reduce  operational  and  support  cost. 

3.  Feedback  from  the  current  maintenance  reporting  systems 
provides  inadequate  and  untimely  data  required  to  improve  reliabil- 
ity of  avionics  equipment. 

4.  Timely  input  from  the  field  concerning  accurate  mission 
profiles  could  enhance  the  reliability  of  avionics  equipment. 

DEFINITIONS 

For  purposes  of  this  study,  the  following  terms  are  defined  to 
clarify  specific  meanings  as  used  in  this  study: 

Corrective  Action:  The  process,  procedure,  or  material  changes 

made  to  correct  the  true  cause  of  a failure,  to  include  documentation. 

Environment : All  the  environmental  stresses  and  combinations 

thereof,  effecting  systems  during  operation  performance,  and/or  field 
environments . 

Failure  Analysis:  The  identification  of  the  failure  mode,  the 

failure  mechanism,  and  the  cause. 

Mean-time-to-repair : The  total  corrective  maintenance  time 

divided  by  the  total  number  of  corrective  maintenance  actions  during 
a given  period  of  time. 
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Maintenance  Reliability:  The  probability  or  probable  duration 

of  specified  performance,  by  a given  system,  subsystem  or  equipment 
and  all  of  its  component  parts,  under  specified  conditions  and 
procedures.  (Readiness  and  ownership  cost  factor;  see  FaiJLure. ) 

Mission  Reliability:  The  probability  that  an  operationally 

ready  system,  subsystem  or  equipment  will  initiate  and  complete  one 
specified  mission  upon  command,  under  specified  conditions  and 
procedures . 

Predicted  Reliability:  That  reliability  which  is  expected  at 

some  future  date,  postulated  on  analysis  of  the  design  and  the 
failure  rates. 

Reliability : The  probability  that  a piece  of  equipment  will 

perform  its  intended  function  for  a specified  period  of  time  under 
stated  conditions. 

Reliability  Assessment:  An  estimate  of  the  achieved  reliability 

obtained  by  calculations  using  data  gathered  during  tests. 

MTBF : Mean  Time  Before  Failure. 

CERT : Combined  Environment  Reliability  Testing. 

Failure  vs.  Relevant  Failure:  No  attempt  will  be  made  to 

clarify  what  constitutes  relevant  failure  vs.  failure.  Existing 
definitions  will  suffice  for  purposes  of  this  study. 


ORGANIZATION  OF  THE  REMAINDER  OF  THE  STUDY 


This  study  is  presented  in  five  chapters.  In  Chapter  T,  the 
problem  is  defined  and  the  purpose  of  the  study  is  outlined. 
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In  Chapter  II,  a review  of  previous  studies  and  related  literature 
is  presented.  The  methods  and  procedures  of  thestudv  are  presented 
in  Chapter  III.  The  analysis  of  the  data  and  the  findings  of  the 
study  are  presented  in  Chapter  IV.  In  Chapter  V,  the  study  is 
summarized  and  significant  conclusions  and  recommendations  are 
reported . 


CHAPTER  II 


REVIEW  OF  LITERATURE 

Recent  studies  of  avionics  equipment  reliability  support  the 
conclusion  that  equipment  is  not  being  designed  for,  or  tested  in, 
an  environment  similar  to  that  in  which  it  will  be  employed. 

As  a result,  operational  stresses  are  causing  failure  modes 

that  were  neither  considered  during  design  nor  discovered  and 

elimin: ted  during  testing.  Of  all  the  challenges  faced  by  avionics 

equipment  designers,  none  exceeds  in  magnitude  or  importance  that 

2 

of  providing  operational  forces  with  reliable  equipment. 

A review  of  the  related  literature  indicated  the  problem  con- 
sidered (within  the  confines  of  this  study)  had  not  previously 
been  investigated.  However,  many  studies  have  been  conducted  on 
the  overall  subject  of  avionics  reliability.  These  studies  con- 
sidered many  factors  which  effect  poor  reliability  of  avionics 
equipment.  These  studies,  ranging  in  duration  of  development  from 
19  to  30  months,  were  conclusive  and  contained  many  excellent 
recommendations.  The  findings  suggestad  there  are  many  inconsis- 
tencies in  policies,  approaches,  etc.,  and  that  there  Was  general 

^A.  Dantowitz,  C.  Hirstchberger , and  D.  Provideo,  Analysis  of 
Aeronautical  Equipment  Environmental  Failure,  Air  Force  Flight 
Dynamics  Laboratory  (Wright  Patterson  AFB : Air  Force  Systems  Com- 

mand), p.  71;  also,  B.  Swett,  "Avionics  Reliability  Study,"  Final 
Report  of  the  Joint  Logistics  Command  Electronic  System  Reliability 
Workshop,  1 October  1975. 

^LTG  Robert  T.  Marsn,  USAF,  "Avionics  Equipment  Reliability, 

An  Elusive  Objective,"  Defense  Management  Journal.  April  1976. 
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confusion  concerning  the  cause  of  poor  reliability  of  avionics 
equipment . 

The  DOD  policy  documents  were  also  included  in  the  review  of 
literature.  The  specific  documents  reviewed  were:  MIL-STD  785-- 

Reliability  Program  Plan;  MIL-STD  756--Reliability  Predictions; 

MIL-STD  781C--Reliability  Design  Qualification  and  Production 
Acceptance  Tests:  Exponential  Distribution;  MIL-HDBK  217; 

AR  702-3--Army  Materiel  Reliability,  Availability  and  Maintain- 
ability; and  DOD  Directive  5000. X (draft) --Reliability  and 
Maintainability  (R&M)  of  Systems  and  Equipment. 

HIGHLIGHTS  OF  PREVIOUS  STUDIES 
Hugnes  Study 

The  Hughes  Aircraft  Company  conducted  a study,  Operational 
Influences  on  Reliability,  for  the  Rome  Air  Development  Center 
(Griffiss  AFB,  NY,  December  1976).  Their  findings  provided  inter- 
esting information  regarding  tne  logistics  problems  associated  wit' 
reliability  and  the  laboratory  demonstrated  reliability  and  the 
laboratory  demonstrated  reliability  results. 

The  objectives  of  this  study  were: 

1.  To  identify  factors  contributing  to  the  reported  differ- 
ences between  the  required,  predicted,  demonstrated,  and  field 
observed  values  of  reliability  (MTBF)  of  avionics  equipment 
currently  operational  in  USAF  aircraft  systems;  and, 

2.  To  develop  prediction  models  which  would  permit  more  valid 
estimates  of  field  operational  reliability. 
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Sixteen  different  avionics  equipment  (52  replaceable  units) 


[ 


used  on  ten  different  USAF  aircraft  types  were  included  in  the 
3 

s tudy . 

Their  findings  indicated  that  many  factors  contributed  to  the 
differences  and  that  there  were  no  simple  explanations  to  account 
for  these  differences . The  true  complexity  of  the  problems  of 
achieving  reliability  in  service  use  v*as  indicated  from  this  study. 

--  The  MTBF  for  avionics  equipment  on  subsonic  bombers  and 
transports  were  two  to  four  times  higher  than  for  similar  equipment 
installed  on  high  performance  tactical  or  training  aircraft. 

--  A base-to-base  comparison  of  three  equipments  on  one  air- 
craft type,  operating  from  nine  different  bases,  revealed  MTBF 
variations  of  as  much  as  five  to  one  from  base-to-base. 

--  The  single  largest  factor  contributing  to  the  discrepancy 
between  either  predicted  or  demonstrated  MTBF  and  field  reported 
MTBF  observed  in  this  study  was  the  use  of  aircraft  flight  hours 
(FH)  for  field  MTBF  assessments  instead  of  equipment  operations 
hours  (OH) . 

— Aircraft  utilization  rates  (flight  hours  per  month  per 
aircraft)  were  observed  to  vary  as  much  as  three  to  one  between 
different  types  of  aircraft.  For  one  equipment  in  the  study  the 
data  indicated  the  40  percent  of  its  reported  failure  had  occurred 
during  nonoperational  periods. 

O 

JHughes  Aircraft  Company,  "Operational  Influences  on  Reliabil- 
ity," Final  Technical  Report,  April  1974-September  1976,  by  Richard 
Baker  and  George  Kern,  RADC-TR-76-366 , December  1976. 

12 


&K-^au» 


— A major  finding  of  the  study  was  that  the  differences  were 
due  to  a combination  of  definitional  factors  (i.e.,  what  is  a 
failure  and  what  is  the  time  base?)  and  operational  factors. 
Significant  differences  in  operational  reliability  were  observed. 

--  A review  of  avionics  maintenance  resources  expenditure 
indicated  that  approximately  39  percent  of  all  maintenance  action 
was  expended  on  the  equipment  interface  and  associated  hardware 
items,  thereby  exposing  the  equipment  to  additional  maintenance 
handling  and  on-off  cycling. 

— The  effect  of  failures  occurring  during  nonoperating  time 
appeared  to  constitute  a significant  percentage  of  the  total  failures 
observed  because  of  the  relatively  low  utilization  rates. 

The  following  recommendations  were  included  in  this  study: 

1.  Establish  more  consistent  measures  of  reliability  assess- 
ment methods . 

2.  Develop  end-user  oriented  measures  of  reliability  and 
maintainability  characteristics. 

3.  Develop  mathematical  models  for  estimating  field  relia- 
bility as  a function  of  demonstrated  and  predicted  reliability. 

Grumman  Study 

A separate  19-month  study  on  "Evaluation  of  Environmental  Pro- 
files for  Reliability  Demonstration"  was  conducted  by  Grumman 
Aerospace  Corp.  for  the  same  Air  Force  Air  Development  Center.^ 

^George  Hirschberger  and  Allan  Dantowitz,  "Evaluation  of  Envir- 
onmental Profiles  for  Reliability  Demonstration,"  Final  Technical 
Report,  June  1973-January  1975,  RADC-TR-7 5-242  , September  1975. 
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The  prtmarv  objectives  were; 

1.  To  determine  the  adequacy  of  the  environmental  profiles  of 
MIL-STD  781  in  simulating  field  stresses. 

2.  Where  inadequacies  existed,  provided  recommended  new  test 
profiles  for  inclusion  in  MIL-STD  781. 

Secondary  objectives  were: 

1.  To  determine  the  adequacy  and  provide  recommendations  for 
improving  demonstration  test  ground  rules  and  scoring  criteria; 
and , 

2.  To  identify  changes  needed  in  reliability  prediction 

methods  to  produce  better  correlation  between  test  and  field 
results.  Their  findings  indicated:  a.  That  the  difference 

between  laboratory  tests  and  field  environmental  exposure  is  one 
of  the  most  significant  reasons  for  avionics  estimated  reliability 
incompatibility;  b.  Increased  environmental  stress  levels  on  hard- 
ware due  to  modern  high  speed,  high  performance  aircraft  are 
responsible  for  many  field  failures;  and  c.  That  almost  50  percent 
of  the  failures  experienced  during  the  study  were  environmentally 
related  (see  Figure  1). 

Ninety-five  line  removal  units  representing  a cross-section 
of  avionic  types  and  applications  were  used  in  this  study.  The 
results  of  the  study  indicated  that  differences  between  laboratory 
demonstrated  and  field  observed  reliability  was  attributable  to  a 
wide  variety  of  factors.  The  variance  factor  contribution  to  the 
reliability  differences  are  reproduced  in  Table  1. 
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AVIONICS  EQUIPMENT  FAILURE  CAUSES 


Figure  1 

Highlights  of  the  study  are  presented  as  follows: 

--  Inconsistent  ground  rules  and  failure  scoring  criteria 
accounted  for  a significant  portion  of  the  difference  between  demon- 
strated and  field  MTBF  , Anomalies  which  should  be  considered 
accountable  are  often  excluded  by  demonstration  test  ground  rules. 

— Current  military  logistic  data  reporting  systems  lack  detail 
in  describing  malfunctions,  resulting  in  counting  incidents  which 
should  be  excluded. 

— Difficulties  in  collecting  equipment  operating  time  data 
resulting  in  flight  time  being  used  on  the  time  base  for  field  MTBF 
determination. 
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TABLE  1 FACTORS  CONTRIBUTING  TO  RELIABILITY  DIFFERENCES 


limited  or  nonexistent. 


--  Reclassification  of  failures,  both  field  and  laboratory. 

--  Fifty  percent  of  the  equipment  studies  had  field  values 
three  times  lower  than  the  corresponding  demonstrated  value. 

— The  reliability  agreement  between  the  laboratory  and  the 
field  was  poorer  on  those  items  that  had  a potential  for  abuse  in 
the  field. 

--  Those  items  that  had  the  more  effective  burn-in  testing  on 
production  units  tended  to  nave  better  reliability  agreements. 

--  There  was  some  evidence  that  the  current  MIL-STD  781  tests 
of  only  requiring  dwells  at  the  temperature  extremes  with  moderate 
rate  of  exchange  between  the  units  is  not  an  adequate  test. 

--  The  vibration  test  duration  was  found  to  be  a poor  repre- 
sentative of  the  accumulated  field  vibration  time.  The  10  minutes 
of  Sinusoidal  vibration  each  hour  at  nonresonant  frequency  between 
20-60  Hz  is  not  representative  of  the  field  environment,  i.e.,  jet 
aircraft,  the  vibration  environment  is  random. 

--  Where  "hi  Rel"  parts  were  used,  the  equipment  had  better 
correlation  between  field  and  demonstrated  reliability. 

— Moisture  was  found  to  be  a major  source  of  field  failure. 

Some  of  the  recommendations  of  the  study  were; 

1.  Revision  of  the  demonstration  test  profile  and  method  of 
profile  construction.  This  would  feature: 

- mission  profile  orientation. 

- variations  in  chamber  temperatures  as  a function  of 
changes  in  flight  conditions  and  thus  approximating  compartment 
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- temperature  rate  of  change  equal  to  flight  levels. 

- coupling  of  cooling  air  variation  with  chamber  tempera- 
ture variations. 

2.  Revision  of  the  demonstrated  test  vibration  profile  to 
feature : 

- random  vibration  for  items  installed  in  jet  aircraft. 

- sinusoidal  sweeps  for  items  installed  in  propeller- 
driven  aircraft. 

- the  vibration  level  to  approximate  those  in  the  field. 

- exposure  to  all  input  frequencies  up  to  2000  Hz. 

3.  Inclusion  of  a humidity  test  that  adequately  reflects 
field  environment. 

4.  Incorporation  of  precisely  defined  failure  criteria  and 
scoring  ground  rules  that  clearly  describe  those  anomalies  to  be 
considered  relevant  and  nonrelevant  as  well  as  provide  the  minimum 
conditions  for  reclassifying  a relevant  failure  as  nonrelevant. 

5.  Include  an  end-items  quality  factor  and  a quality  assur- 
ance factor  in  the  prediction  technique. 

6.  Mission  profile  data  should  be  developed  for  those  items 
where  this  information  is  not  known. 

National  Academy  of  Science  Study 

The  Air  Force  Systems  Command  sponsored  a study  on  Reliability 
in  Aeronautical  Avionics  Equipment.  1975.^  This  study  is  in  a 

^National  Research  Council,  Reliability  in  Aeronautical  Avionics 
Equ ipment . A Report  of  the  Air  Force  Technological  Tradeoffs  Panel 
to  the  Air  Force  Studies  Board,  1975. 
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report  of  the  Air  Force  Technological  Tradeoffs  Panel.  The  concern 
of  this  panel  was  the  cost  of  ownership  of  new  weapons  systems  and 
the  question  of  the  relative  cost-effectiveness  of  different  tech- 
nologies in  achieving  the  same  or  overlapping  military  objectives. 
They  pointed  out  that  the  lack  of  reliability  and  poor  maintaina- 
bility contributed  to  the  cost  of  ownership  in  three  ways: 

1.  The  direct  cost  of  maintaining  the  unreliable  equipment-- 
in  labor,  personnel  training,  spares,  replacement  costs,  and 
logistics  overhead. 

2.  Indirect  cost  of  down-time. 

3.  Indirect  cost  of  failure  during  a mission,  including  the 
question  of  pilot  safety. 

The  study  identified  a need  for  more  realistic  and  effective 
procedures  for  acquiring  new  systems,  especially  in  a technology 
changing  as  fast  as  avionics  is  today.  A new  approach  would  be  to 
identify  the  first  issuance  of  a Required  Operational  Capability 
(ROC)  as  a set  of  objectives.  These  objectives  should  be  altered 
and  improved  by  an  iterative  process  extending  well  into  the 
development  phase  in  which  feedback  plays  a central  role.  This 
approach  illustrates  a central  point  about  reliability  which  is  too 
often  forgotten.  The  point  is  the  achievement  of  reliability  in 
systems  under  development  depends  to  a large  degree  upon  the  pro- 
cessing of  information  about  the  specific  systems  involved,  and  it 
is  very  difficult  to  obtain  feedback  data  about  the  systems  until 
the  systems  are  developed.  A solution  to  this  problem  depends  upon 
a very  rigSrous  data-gathering  effort  starting  at  the  earliest  stage 
of  the  development  process. 
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Some  of  the  specific  recommendations  of  the  study  were; 

--  There  is  a need  for  more  failure  analysis  and  more  emphasis 
on  reliability  by  the  program  managers. 

--  The  test  programs  need  to  be  restructured  to  provide  more 
realistic  tests  that  approximate  the  field  environment. 

--  Conduct  earlier  failure  analysis  and  subsystems  tests  which 
would  provide  information  on  failure  modes  that  can  be  corrected  at 
minimum  cost. 

— There  should  be  more  post-deployment  efforts  to  improve 
avionic  reliability. 


RAND  Study 

Mclver  and  Robinson  prepared  a report , A Proposed  Strategy  for 
the  Acquisition  of  Avionics  Equipment,  which  was  the  result  of  a 
Rand  Studies  sponsored  by  the  USAF  rn  December  1974. ^ 

Their  purpose  was  to  develop  a strategy  for  the  acquisition  of 
avionics  equipment  that  would  improve  operational  reliability  and 
reduce  life-cycle  cost. 

The  conclusions  and  recommendations  of  this  study  were: 

— An  alternative  procurement  policy  and  management  structure 
for  integrated  avionics  systems. 

— An  alternative  procedure  for  avionics  development. 

— The  establishment  of  "Air  Force  Avionics  Development  Center." 


£ 

D.  W.  Mclver,  A.  I.  Robinson,  H.  L.  Shulman  with  the  Assist- 
ance of  W.  H.  Ware,  A Proposed  Strategy  for  the  Acquisition  of 
Avionics  Equipment,  R-1499-PR,  December  1974,  a report  prepared  for 
United  States  Air  Force  Project  Rand. 


20 


The  rationale  for  the  first  recommendation  was  that  at  the  time 
of  the  report  it  was  suggested  that  the  Air  Force  had  neither  the 
appropriate  policies  nor  the  management  structure  necessary  for  the 
acquisition  of  accurate  reliable  equipment.  The  integration  of 
the  various  avionic  subsystems  was  perceived  as  a major  problem. 

Recommendations  made  suggested  the  need  for  a new  development 
approach  which  would  take  into  consideration  the  factors  shown  in 
schematic  no.  1 during  the  acquisition  cycle.  (See  schematic  no.  1, 
next  page . ) 

The  third  recommendation  had  as  its  rationale  the  need  for  a 
separate  AF  Avionics  Development  Center  with  a strong  civilian 
component.  This  concept  included  manager  organization  and  tech- 
nical expertise  specifically  intended  for  avionics  development  and 
its  associated  reliability  problem. 

The  civilian  component  would  provide  continuity,  corporate 
memory,  and  would  have  a continuing  responsibility  for  the 
performance  of  each  particular  piece  of  equipment. 

General  Electric  Report 

Selby  and  Miller  presented  a paper  "Reliability  Planning  and 
Management  (RPM)"^  at  the  ASQC/SRE  Seminar  in  1970,  which  addressed 
the  credibility  in  avionics  reliability  planning  and  acquisition. 
"Experience  has  demonstrated  that  to  a varying  degree,  a reliability 

^J.  D.  Selby  and  S.  G.  Miller,  "Reliability  Planning  and 
Management  (RPM),"  a paper  presented  at  ASQC/SRE  Seminar,  Niagara 
Falls,  NY,  September  26,  1970. 
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Schematic  No. 


program  credibility  gap  exists  between  stated  equipment  reliability 
requirements  and  realized  achievement."1  They  suggested  the  gap 
stems  largely  from  the  lack  of  a uniform  method  of  reliability  pro- 
gram structuring  and  evaluation.  This  in  turn  results  in  the 
implementation  of  unrealistic  programs  with  unachievable  require- 
ments. Currently,  both  the  contractor  proposal  manager  and  the 
equipment  buyer  lack  tools  with  which  to  evaluate  with  reasonable 
conviction  that  any  of  the  proposed  programs  can  result  in  a product 
compliant  with  the  reliability  requirements. 

A review  of  the  performance  of  avionics  suppliers  indicated 

that  for  the  newly  developed  equipments  in  the  late  1960s,  the 

initial  average  achieved  reliability  was  approximately  eight 

percent  of  their  specified  requirements.  The  problem  remains  that 

"management,  government  and  industry  lack  a quantitative  yardstick 

with  which  to  dimension,  plan,  manage,  fund,  and  monitor  reliabil- 

9 

ity  development  as  a part  of  equipment  development."  Recognizing 
the  past  difficulties  and  lack  of  uniform  success,  it  is  mandatory 
that  a change  in  the  practices  of  Reliability  Planning  and  Manage- 
ment be  implemented.  The  authors  suggested  the  development  of  a 
methodology  which  has  the  credibility  of  an  exacting  science, 
incorporates  the  flexibility  for  alternative  program  planning,  and 
contains  standardization  factors  to  facilitate  a uniform  evaluation 
structure  suitable  for  application  by  both  contractor  and  buyer. 

^Ibid  . , p . 1 . 

^ Ibid . , p . 2 . 


The  methodology  designed  by  these  authors  called  Reliability 
Planning  and  Management  (RFM)  reduces  the  planning  and  resource 
allocation  requirements  into  a simple,  guaranteed,  and  objectively 
usable  format.  The  following  factors  are  basic  to  this  approach: 

1.  No  laws  of  physics  are  violated  by  the  designer  and  no 
beyond-the-state-of-the-art  requirements  are  imposed. 

2.  The  criteria  to  be  fulfilled  consisted  of  prediction  versus 
requirement,  initial  product  capability  assessment,  reliability 
growth  rate,  product  experience  gained  through  extended  environ- 
mental exposure,  calendar  time,  and  change  constraints. 

3.  Successful  implementation  requires  that  specific  compliance 
be  achieved  for  each  criterion — Prediction  versus  Requirements, 
Product  Capability,  Reliability  Rate  of  Growth,  and  Product 
Evaluation  Exposure. 

This  approach,  primarily  a management  tool,  was  designed  to 
bridge  the  gap  between  stated  reliability  requirements  and  imple- 
mentation planning.  It  is  applicable  to  establishing  plans,  pro- 
jecting effort,  evaluating  proposals  and  monitoring  performance. 

It  may  be  used  equally  by  buyer  and  contractor. 

SUMMARY 

The  highlights  and  findings  of  these  studies  indicate  that  the 
reliability  of  avionics  is  influenced  by  many  factors  as  cited 
throughout  these  previous  studies.  And  it's  quite  a challenge  to 
improve  the  reliability  of  the  complex  systems  and  subsystems  given 
the  current  budget  constraints. 
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The  emphasis  on  reliability  is  not  new,  emphasis  on  field 
reliability  rather  than  statistical  prediction  computed  for  sub- 
system and  system  reliability  values  is  a major  new  thrust  requiring 
explicit  attention  for  each  program.  Emphasis  on  field  reliability 
focuses  attention  on  performing  more  realistic  tests  to  verify  that 
the  new  systems  and  subsystems  will  perform  reliably  and  can  be 
supported  in  the  field  environment. 

Most  of  the  DOD  Directives  and  MIL-STDs  on  reliability  refer- 
enced previously  are  in  the  process  of  being  revised.  The  focus 
now  is  to  provide  increased  test  realism  and  closer  correlation 
between  field  reliability  and  the  laboratory  demonstrated  relia- 
bility. MIL-STD  781C  Reliability  Design  Qualification  and 
Productive  Tests  Exponential  distribution  is  an  attempt  to 
accomplish  this  objective. 
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CHAPTER  III 

METHODOLOGY  OF  THE  STUDY 

This  chapter  will  explain  the  method  of  selecting  participants, 
the  instrument  utilized  in  data  collection,  and  the  method  used  in 
conducting  the  interview  sessions. 

PURPOSE  OF  THE  STUDY 

The  purpose  of  this  study  was  to  explore  reliability  problems 
associated  with  engineering  development  (Tes t-Analyze-Find  and 
Correct  Phase)  of  avionics  systems  and  subsystems;  to  examine  some 
of  the  implications  of  the  C version  of  MIL-STD  781;  to  identify 
some  of  the  significant  issues  and  problems  associated  with  achiev- 
ing reliability  for  avionic  subsystems;  and  to  provide  recommenda- 
tions that  may  result  in  improved  reliability  and  contribute  to 
reduction  in  the  cost  of  ownership. 

SELECTION  OF  THE  PARTICIPANTS 


The  participants  included  31  persons  within  the  Department  of 
Defense,  industry,  and  the  academic  community  who  are  involved  in 
various  aspects  of  avionics  subsystems  reliability. 

The  criteria  for  selecting  the  participants  was  based  upon 
their  being  identified  as  authorities  in  the  field  and  the  recom- 
mendations from  leaders  in  each  area,  i.e.,  DOD,  industry,  and  the 
academic  community.  The  numbers  of  participants  were  limited  by 
the  time  constraints  for  conducting  the  study. 
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DESIGN  OF  THE  INSTRUMENT 


The  structured  interview  method  of  data  collection  was  used  in 
this  study.  The  structured  interview: 

1.  Increases  the  likelihood  of  obtaining  more  accurate 
informat  ion . 

2.  Allows  for  clarification  of  ambiguous  statements  and  col- 
lection of  supplementary  information. 

3.  Enhances  the  recall  of  relevant  material. 

In  order  to  obtain  valid  responses  which  reflected  the  intent 
of  the  research  undertaken,  the  structured  interview  method  of  data 
collection  was  the  most  acceptable  approach. 

The  following  questions  were  used  during  the  interview  sessions: 

1.  What  is  your  perception  of  the  current  DOD  reliability  test 
philosophy? 

2.  Do  you  think  DOD  has  a common  definition  of  what  consti- 
tutes a reliable  system,  i.e.,  field  reliability  to  the  logistician 
may  not  mean  the  same  thing  to  the  designer  or  contract  administrator? 

3.  Does  the  need  exist  for  DOD  to  provide  more  guidance  or  a 
policy  directive  to  establish  a more  realistic  reliability  program? 

4.  What  factors,  other  than  environmental,  influence  poor 
reliability  in  the  field? 

5.  In  your  opinion,  do  we  adequately  monitor  those  factors 
that  effect  equipment  reliability? 


^D.  Miller,  Handbook 
ment . 1970,  p.  86. 


for  Research  Design  and  Social  Measure- 
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6.  Can  the  reliability  of  avionic  subsystems  be  improved  by 
improving  the  reliability  test  methodology  during  the  early  phase 
of  the  acquisition  cycle? 

7.  Do  we  do  an  adequate  job,  contractually,  defining  relia- 
bility requirements  (for  avionic  subsystems)? 

8.  Does  the  current  logistic  reporting  system  provide  the 
kind  of  data  from  the  field  necessary  to  improve  the  reliability 
of  avionics  subsystems? 

9.  Does  the  DOD  representative  provide  the  equipment  designer 
and  the  reliability  engineer  adequate  information  during  the  initial 
phase  of  the  program? 

10.  What  is  your  perception  of  our  ability  to  provide  the 
contractors  with  realistic  mission  profiles? 

11.  How  much  reliability  consideration  should  be  given  to  the 
conceptual  and  initial  design  phase? 

12.  How  much  consideration  should  be  given  to  the  reliability 
requirements  during  engineering  development  testing? 

13.  What  should  be  the  role  of  the  Government  representative 
during  the  engineering  development  test,  i.e.,  test  find  and  fix? 

14.  What  are  your  perceptions  of  the  cost  impact  of  imple- 
menting the  C version  of  MIL-STD  781? 

15.  If  we  implement  the  C version  of  MIL-STD  781,  will  we 
have  closer  correlations  between  lab  test  and  field  reliability 
data? 
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These  questions  were  developed  from  preliminary  interviews 
with  experts  in  the  field,  based  upon  their  experiences  and  reflec- 
tive of  their  perceptions  of  problems,  issues,  and  solutions  to  the 
problems  of  avionics  subsystems  reliability. 

COLLECTION  OF  THE  DATA 
Initial  Contacts 

The  researcher  contacted  key  representatives  with  the  DOD  agen- 
cies, military  test  facilities  managers,  managers  of  reliability 
test  laboratory  in  private  industry,  a West  Coast  University  (the 
Graduate  School  for  Engineering  Test  and  Evaluation),  and  profes- 
sional societies  that  have  a major  interest  or  responsibility  for 
the  reliability  of  avionics  systems  and  subsystems. 

These  initial  telephone  contacts  were  made  to  discuss  the 
objectives,  time  frame,  and  scope  of  the  study;  to  explore  areas 
of  common  concern;  and  to  secure  appointments  for  formal  interview 
sessions.  The  content  of  these  initial  contacts  was  the  basis  for 
development  of  the  15  questions  used  as  the  interview  guide. 

Copies  of  the  formal  proposal  were  mailed  to  each  participant. 

The  Interview  Sessions 

A complete  explanation  of  the  study  was  given  to  all  partici- 
pants at  the  beginning  of  the  interview  session. 

The  participants  were  interviewed  in  their  offices  and/or  lab 
at  their  respective  agency,  during  normal  working  hours.  Those 
participants  representing  tne  professional  organizations  were 
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interviewed  while  attending  the  "Reliability  Conference"  held  by 
the  Institute  of  Environmental  Science,  Los  Angeles,  California. 


The  length  of  each  interview  was  an  average  of  four  hours . In 
some  instances,  the  participants  also  responded  in  writing  to  the 
15  questions  used  as  the  interview  guide.  Without  exception,  all 
of  the  participants  were  highly  interested  in  the  subject. 

Follow-up  Contacts 

Follow-up  contacts  were  made  with  participants  for  further 
clarification  as  needed  and/or  for  complementary  data. 

TREATMENT  OF  THE  DATA 

The  data  gathered  from  the  interviews  was  analyzed  by  the 
researcher  to  document  the  various  perceptions  of  the  problems 
associated  with  improving  reliability  of  avionics  equipment,  and  to 
identify  test  methodologies  and  ways  of  reducing  costs  of  testing 
or  methods  of  improving  reliability  of  avionics  subsystems  in  the 
field. 

A copy  of  this  study  will  be  provided  to  each  agency  partici- 
pating and  a copy  to  the  Office  of  Defense  Research  and  Engineering 
per  their  request. 
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CHAPTER  IV 


FINDINGS  AND  ANALYSIS  OF  THE  DATA 

This  chapter  reports  the  findings  of  the  study.  It  includes  a 
review  of  the  data  collection  method,  a discussion  of  the  findings, 
and  a listing  of  the  participants  (Figure  2),  and  a listing  of  the 
distribution  of  responses  by  agency  (Figure  3). 

REVIEW  OF  THE  DATA  COLLECTION  METHOD 

The  personal  structured  interview  metnod  was  used  to  collect 
the  data.  This  approach  provided  the  interviewer  an  opportunity  to 
collect  supplementary  information  about  the  respondent's  professional 
expertise  and  his  work  environment.  This  added  information  enhanced 
the  interpretation  of  the  data  and  facilitated  obtaining  more 
accurate  information  than  would  have  been  by  any  other  technique. 

DISCUSSION  OF  THE  FINDINGS 

The  findings  of  the  study  are  presented  as  follows:  a discus- 

sion of  the  perceptions  held  by  the  various  experts  on  the  subject 
of  avionics  reliability;  a discussion  of  each  question  based  on  the 
responses  obtained;  and  a summary  of  the  data. 

The  various  perceptions  held  by  the  participants  on  avionics 
reliability  are  noted  as  they  related  to  the  study.  One  part  of 
the  objective  focused  on  the  identification  of  significant  issues 
and  problems  associated  with  achieving  reliability  for  avionics 
systems  and  subsystems. 
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The  following  persons  participated  ir  the  structured 
interviews  for  this  study: 


Agency 


R L Baker , Manager 
J C Bear,  Manager,  Reliability 
Lab 

R A Blondin,  Statistician 

D A Bond,  Manager 

L K Brooks 

Dr  A Burkhard 

Dr  B Campbell,  Professor 

LTC  S Dizek 

R E Eckel , Manager 

J L Gregson 

S Grubman 

Dr  J Guarrera,  Director  of 
Research 

P G Halamandaris , VP  of 
Engineering 
W A Harmon,  Jr 
J Hess 
B Jones 

G A Kern,  Manager 

J P neslie.  Manager,  Reliabili 

MAJ  P A Me Ad am 

LTC  T A Musson 

A Nordstrom 

A Pollack 

Dr  Charles  °anders, 

Dean  of  Engineering 
W C Savage,  Lab  Manager 

R K Shinkle 
D Sinback 
COL  B Swett 
W Wallace , Jr 
C Wigginton 
LTC  W D Wilson 
M Zak 


Hughes  Aircraft  Company 
General  Dynamics , Pomona 

Naval  Air  Facility 
Environmental  Lab,  Northrop  Corp. 

Naval  Air  Systems  Command 
AFFDL/FEE  AFB 

California  State  University,  Nort.hridge 
ASD/RAOE 

Naval  Avionics  Facility 
Naval  Avionics  Facility 
US  Army  Electronics  Command 
California  State  University, 

Northridge 
GD  Electronics 

ASD/ENES , Wright  Patterson  AFB 

Cdr  DARCOM 

ASD/ENO 

Hughes  Aircraft  Company 
ty /Texas  Instruments,  Inc 

ASo/RAOE,  Wright  Patterson  AFB 

HQ  DA  USAF , Pentagon 

Cdr , DARCOM 

HO DA,  Pentagon 

California  State  University, 

Northriage 

AFFDL/FEE,  W.rgat  Patters.-n  AFB 
Naval  Avionics  Facility 
Naval  Air  Systems  Command 
ODD  RA-E,  Pentagon 
NAVELEX 

Naval  Air  Systems  Command 

HQDA,  Pentagon 

US  Army,  Electronics  Command 


Figure  2 


Structured  Interviews  Participants 
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Distribution  of  Response  by  Organization 
and  position 


Organization 
N = 31 

Response 

Percentage  of 
Total  Response 

US  Army 

Electron  Command 

2 

DARCOM 

2 

HQ  DA  ODCSDA 

2 

6 

19.3% 

US  Air  Force 
Wright-Patterson  AFB 

asd/eno 

2 

affdl/fee 

2 

asd/enes 

2 

The  pentagon 

HQ  USAF  AF/LGYE 

1 

DOD  R&E  (Planning) 

1 

8 

2 5.8% 

US  Navy 

NAFI 

4 

NAVAIR 

3 

NAVELEC 

1 

8 

25.8% 

Industry 

General  Dynamics  Electronics 

1 

General  Dynamics,  Pomona 

1 

Texas  Instruments,  Inc. 

1 

Northrop  Corporation 

1 

Hughes  Corporation 

2 

6 

19.3% 

Academic  Community 

California  State,  Northridge 

3 

9.6% 

Figure  3 
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DISCUSSION  OF  PARTICIPANT  RESPONSES  RE  AVIONICS  RELIABILITY 

The  most  frequently  cited  issues  and  problems  associated  with 
achieving  reliability  for  avionics  equipment,  as  perceived  by  trie 
participants,  are  identified  as  follows: 

1.  The  methods  of  comparing  reliability  parameters  with 
performance  parameters  (for  trade-off)  during  the  acquisition  pro- 
cess was  considered  inadequate,  i.e.,  cost,  weight,  performance  are 
easily  quantified  and  measured,  whereas  reliability  penalties  have 
minimal  impact  until  the  equipment  is  deployed. 

2.  The  existing  reliability  requirements  are  not  enforced  for 

various  reasons.  Examples  cited  included:  cost  and  schedule. 

Reliability  development  tests  are  often  squeezed  out  of  the 
acquisition  program  schedule. 

3.  There  exists  a lack  of  clearly  defined  reliability  require- 
ments and  a lack  of  emphasis  on  reliability  during  the  conceptual 
and  design  phase.  Some  of  the  responses  indicated  that  the  DOD 
representatives  do  not  adequately  define  the  requirements  contractu- 
ally; reliability  emphasis  should  start  at  the  conceptual  phase  and 
continue  throughout  the  life  of  the  equipment. 

4.  Inadequate  funds  prevent  DOD  reliability  representatives 
from  participating  during  the  testing  cycle  and  from  observing  the 
contractor  reliability  techniques. 

5.  Inadequate  fault  isolation  techniques  were  considered  a 
major  issue  because  of  the  introduction  of  new  automatic  test 
equipment  and  software  problems  related  to  that  equipment. 

6.  The  maintenance  reporting  system  does  not  provide  the  feed- 
back required  to  improve  systems  reliability,  i.e.,  the  typical 
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reporting  systems  operation  time  is  used  as  the  basis  for 
calculating  equipment  failure  performance.  In  aircraft  the 
equipment  o^«irating  time  accumulated  ground  checkout  is  not 
considered.  Other  problems  associated  with  the  reporting  system 
are  failure  definitions,  and  unscheduled  calibration  and  mainten- 
ance which  are  often  counted  as  failures. 

7.  Subsystems  interface  effects  which  causes  other  equipment 
failure . 

8.  Software  programs  which  produce  failure  or  tne  appearance 
of  failure  were  also  considered  a significant  problem. 

9.  A very  large  percentage  of  failures  are  caused  by  improper 
maintenance. 

10.  Increased  sophistication  of  avionic  equipment  requirements 
and  the  rate  of  change  in  technology  were  among  failure  identifi- 
cations listed. 

11.  The  acquisition  process  and  management  environment  was 
identified  as  part  of  the  problem.  It  was  suggested  that  a 
shortened  procurement  cycle  for  avionic  equipment  might  improve 
reliability . 

12.  Other  issues  and/or  problems  identified  by  the  partici- 
pants included:  increased  system  complexity,  poor  management  prac- 

tices, unrealistic  test  conditions,  inadequate  environmental 
information,  and  inadequate  failure  analysis. 

These  problems  and/or  issues  identified  by  the  participants 
are  in  addition  to  those  discussed  in  Chapter  II.  The  participants 
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concurred  that  the  issues  addressed  in  previous  studies  also 
influenced  poor  reliability. 

The  second  part  of  the  objective  of  this  study  was  to  examine 
some  of  the  implication  of  implementing  MIL-STD  781C. 

A brief  synopsis  is  provided  for  those  who  have  not  had  an 
opportunity  to  review  the  c version  of  the  781.  This  version 
represents  a complete  revision  of  MIL-STD  781B.  The  scope  of  the 
specification  has  been  broadened  and  the  environments  are  more 
severe  than  previously  required  in  the  B version. 

Some  of  the  significant  changes  of  the  C version  are: 

--  Combined  Environment . This  approach  is  believed  to  provide 
more  accurately  simulated  operational  conditions.  The  combined 
stress  approach  (Temperature,  vibration,  Attitude,  and  Moisture) 
should  demonstrate  reliability  values  which  are  in  closer  agreement 
with  the  values  observed  in  the  field. 

--  Integrated  Test  Planning.  More  emphasis  will  be  placed  on 
integrated  test  planning  (Table  2,  Integrated  Test  Program). 

--  Data  Evaluation.  A new  paragraph  on  data  evaluation  will  be 
added  to  assist  in  analyzing  results  of  tests. 

--  Stress  Levels  have  been  increased. 

--  Definitions  have  been  refined.  Pattern  failures  have  been 
refined . 

The  lower  and  upper  limits  mean  time  before  failures  have  been 
redefined  to  lower  and  upper  test  between  failures.  The  upper  and 
lower  limits  are  to  be  called  out  in  section  3 and  4 of  individual 
equipment  specifications.  Design  ratio  has  been  changed  back  to 
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(1)  TEST,  ANALYZE  AND  FIX 

(2)  SPECIFIED  MTBF  DEMONSTRATION  PER  SPECIFIED  TEST  PLAN  AND  ENVIRONMENTS 

(3) -  USUALLY  LESS  STRINGENT  STATISTICALLY  THAN  RELIABILITY  QUALIFICATION  TESTS 

(A)  & (B)  ALTERNATIVE  TEST  PLANS,  (B)  REQUIRES  USE  OF  TAAF,  (A)  AS  SPECIFIED  BY  MIL-STD-781C 
NOTE:  IF  (1)  TAAF  TYPE  TEST  IS  NOT  DONE,  (2)  REL.  QUAL.  TEST  MUST  BE  DONE  ON  PROTOTYPE  UNITS 


discrimination  ratio  with  basically  the  same  definition.  The 


mission  profile  definition  has  been  completely  reworded  to  emphasize 
both  span  time  or  events  and  expected  environmental  conditions. 

A significant  amount  of  coordinated  efforts  took  place  in  the 
development  of  MIL-STD  781C.  A series  of  meetings  were  held:  (1) 

within  each  command  of  the  services,  (2)  a one-week  tri-service 
meeting,  (3)  a one-week  meeting  between  the  representatives  of  the 
services  and  industry,  and  (4)  meetings  with  the  following  profes- 
sional societies--Institute  of  Environmental  Science  (IES),  Ameri- 
can Society  of  Quality  Control  (ASQC) , National  Security  Industrial 
Association  (NSIA) , Quality  & Reliability  Assurance  Advisory 
Committee  (QRAAC) . 

Each  professional  society  represented  many  companies  and  a 
wide  variety  of  disciplines.  For  example,  the  IES  input  represented 
22  separate  companies  and  six  major  disciplines  within  eacn  organi- 
zation participating  in  reliability  development  (Figure  4). 

A summary  of  the  perceptions  held  by  the  participants  regard- 
ing the  implications  of  781C  are  as  follows: 

--  The  cost  impact  of  implementing  the  c version  as  now  pro- 
posed (30  Aug  76  draft)  would  be  astronomical.  The  nonrecurring 
cost  would  be  high  because  of  the  requirement  for  new  test  equipment 
and  additional  environmental  facilities.  The  recurring  cost  would 
also  be  higher  because  of  added  maintenance  cost  coupled  with  the 
need  for  additional  controls. 

--  Concern  was  expressed  that  this  would  complicate  fault 
isolation.  When  a failure  occurs,  during  a combined  environment 


test,  how  do  you  isolate  the  specific  environment  that  caused  the 
failure,  how  much  time  is  required  to  make  that  determination? 

--  781C  was  viewed  by  some  as  an  unnecessary  expense  and 
unaffordable  in  the  near  time  frame.  There  were  those  who  believed 
that  major  improvements  could  be  achieved  at  lower  cost  by  enforc- 
ing existing  specif icatiore  and  improving  reliability  managerial 
practices . 

— The  cost  of  testing  would  be  increased  maybe  to  the  point 
that  the  program  manager  and  other  decision-makers  would  be  encour- 
aged to  cut  out  some  of  the  tests;  thus  less  testing  would  be 
accomplished  than  is  now  being  accomplished. 

--  There  would  be  no  significant  changes  in  field  reliability 
as  a result  of  these  revisions  because  of  the  many  other  factors 
which  effect  reliability  that  were  not  addressed  in  the  specifica- 
tion, i.e.,  training  of  maintenance  personnel,  improper  maintenance, 
the  maintenance  reporting  cycle,  human  engineering,  etc. 

--  On  the  positive  side  many  favored  the  release  and  field 
trials  of  781C.  They  believed  the  document  would  result  in  improved 
field  reliability.  The  more  realistic  test  conditions  would  close 
the  disparity  between  lab  and  field  reliability  assessment,  and  the 
new  specifications  would  place  greater  emphasis  on  reliability 
attainment . 

--  Most  agreed  that  the  test  cost  would  increase;  however,  it 
was  believed  that  the  long-range  cost  of  ownership  would  be 
decreased  because  of  improved  reliability. 
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— MIL-STD  781c  would  have  a positive  impact  on  reliability 
development  procedures  by  forcing  communications  between  environ- 


mental and  reliability  organizations,  both  in  government  and 
industry.  It  would  force  all  engineering  disciplines  that  should  be 
involved  in  reliability  development  to  recognize  their  participative 
roles . 

--  Most  of  the  participants  agreed  that  781b  did  not  impose  a 
realistic  and/or  adequate  vibration  environment  for  high  performance 
avionics  equipment.  The  new  test  levels  must  be  reasonable  and 
based  on  real  requirements,  i.e.,  they  should  be  realistic  to 
include  mission  profile  for  specific  aircraft  and  possibly  location 
within  the  aircraft. 

The  integrated  test  approach  outlined  in  MIL-STD  781C  was 
viewed  by  some  as  a positive  step  toward  achieving  the  DOD  goal  of 
reliability  improvement  of  weapons  systems  (See  Table  2,  Integrated 
Test  Program,  from  the  August  draft  of  MIL-STD  781C).  (See  p.  38.) 

The  third  part  and  the  primary  focus  of  the  study  was  to  explore 
the  reliability  problems  associated  with  engineering  development 
(Test-Analyze-Find  and  Correct  Phase)  testing  of  avionics  subsystems 
and  equipment;  and  to  attempt  to  provide  recommendations  that  may 
result  in  improved  reliability  and  contribute  to  reduction  in  the 
cost  of  ownership. 


The  materiel  acquisition  process  and  the  total  managerial 
environment  was  frequently  cited  as  part  of  the  problems  associated 
with  avionics  reliability.  A brief  review  of  these  two  areas  is 


presented  below.  The  basic  acquisition  process  is  a sequence  of 


specific  phases  of  program  activity  and  decision  points. 


An  overview  of  the  key  phases  and  decision  points  is  presented 
in  Chart  1,  System  Acquisition  Model.  Each  phase  of  the  process 
has  as  its  underlying  objective  the  progressive  refinement  and 
quantification  of  the  technical,  economic,  and  schedule  projections 
that  are  the  basis  for  system  requirement. 

Chart  2 is  representative  of  the  management  environment  asso- 
ciated with  the  procurement  process.  This  specific  chart  repre- 
sents the  Army's  management  structure.  Each  of  the  services  has  a 
similar  structure. 

Engineering  development  is  a part  of  this  total  acquisition 
process.  Engineering  development  testing  and  planning  as  a part  of 
the  phase  should  be  continuous  throughout  the  development  cycle. 

The  purpose  of  Engineering  Development  Testing  is  to  reduce  risks 
as  early  as  possible.  The  participants  agreed  that  it  should 
include  laboratory  tests,  operational  testing  and  systems  and 
‘ subsystems  demonstrations.  The  planning  for  these  tests  should  be 
as  early  as  possible  in  the  System  Acquisition  Phase.  The  program 
manager  must  evaluate  the  tactical  importance  of  the  specific 
requirements  relative  to  the  design  state-of-the-art,  program 
scheduling,  funding  constraints,  availability  of  test  facilities 
and  other  factors  before  he  can  define  the  specific  test  require- 
ments. He  also  determines  the  reliability  failure  characteristics 
before  selecting  the  particular  test  method.  An  example  of  a 
nominal  development  phasing  is  shown  in  Figure  5.  This  nominal 
development  phasing  shows  Reliability  Demonstration  Growth  Test 
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at  the  end  of  development  and  operation  11  tests.  It  is  the  position 
of  this  researcher  that  reliability  planning  should  start  at  the 
Requirement  Concept  Generations  Phase  and  that  informal  tests  to 
identify  failure  should  start  as  soon  as  you  have  hardware.  The 
Engineering  Development  Test  program  should  be  designed  to  reduce 
risk  as  early  as  possible,  to  demonstrate  that  the  Engineering 
design  and  development  process  is  complete,  to  verify  the  attain- 
ment of  technical  performance  specifications  and  objectives,  and  to 
demonstrate  that  the  design  risks  have  been  minimized. 

Most  of  the  participants  agreed  that  the  best  time  to  improve 
reliability  is  during  the  Engineering  Development  test.  Unfortun- 
ately the  typical  situation  is  that  the  customer  has  not  had  the 
budget  and  time  required  to  do  the  development  testing  properly. 

The  contractors  are  asked  to  build  a flyable  prototype  which  has 
poorly  defined  reliability  requirements.  Then,  if  the  prototype 
is  successful,  production  units  are  procured  on  an  FFP  contract  in 
the  shortest  time  and  lowest  cost  possible.  This  approach  does 
not  support  the  overall  DOD  reliability  goal. 

During  engineering  development  testing  the  following  should  be 
major  considerations: 

— Are  the  systems  and  subsystems  requirements  detailed  to  the 
ROC?  Does  the  design  approach  take  into  consideration  reliability 
requirements  and  supportability? 

--  Have  the  MTBF  been  specified,  and  is  it  based  on  realistic 
requirements? 

— Are  there  communication  channels  for  resolving  unrealistic 
requirements  ? 
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--  Have  the  MTTR  been  established? 

--  Have  contracts  been  established  for  technical  performance 
measurement,  failure  modes  and  effective  analysis  and  integrated 
logistic  support? 

--  Are  funds  and  schedules  adequate  to  conduct  those  tests 
necessary  to  test-analyze-f ind  and  correct  differences? 

--  During  the  Engineering  Development  Phase,  the  reliability 
growth  plan  should  be  a major  consideration.  The  reliability  growth 
management  plan  should  include  as  a minimum  these  elements: 

1.  A description  of  the  sources  of  historical  data  used  in 
preparing  the  planned  growth  curves. 

2.  The  Engineering  rationale  used  in  deriving  the  planned 
growth  curves . 

3.  A process  for  identification  and  elimination  of  failure 
modes,  including  the  resources  to  be  used  in  these  activities. 

4.  Systems  and  critical  subsystems  planned  growth  curves. 

5.  A method  for  assessing  reliability  at  successive  points  in 
time  and  comparing  these  assessments  with  the  planned  growth  curves. 

Most  of  the  personnel  interviewed  agreed  that  Environmental 
Development  Testing  Phase  should  include  sufficient  tests  to  provide 
confidence  that  the  equipment  will  operate  satisfactorily  in  the 
intended  use  environment.  These  tests  should  also  include  0 expo- 
sure to  individual  environments  and  combinations  of  environments 
where  it  is  economically  feasible.  The  participants  also  agreed 
that  the  reliability  testing  during  the  Engineering  Development 
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the  equipment  meets  or  exceeds  the  theta  0.  A test-analyze-f ind 
and  correct  approach  should  be  used  to  analyze  all  failures 
incorporating  corrective  action  and  restating.  During  this  period 
the  government  representative  should  be  fully  involved  with  the 
contractor  in  attempting  to  resolve  problems--not  counting  failures. 

During  the  researcher's  visit  to  Wright-Patterson  AFB,  Flight 
Test  Lab,  several  hours  were  spent  observing  tests  in  progress  and 
discussing  the  overall  Combined  Environmental  Reliability  Test 
(CERT)  evaluation  program.  This  program  consists  of  a series  of 
tests  of  avionics  equipment  which  evaluates  the  technical  merit  of 
conducting  reliability  tests  with  combined  environments.  These 
environments  included  random  vibration,  humidity,  temperature, 
altitude,  and  air  flow.  The  Combined  Environment  Reliability 
Evaluation  Program  is  divided  into  three  phases: 

1.  Establish  technical  merit  of  CERT. 

2.  Evaluate  cost  effectiveness  of  CERT. 

3.  Implementation  planning. 

The  facility  used  at  Wright-Patterson  AFB  is  a special  design  facil- 
ity. The  design,  fabrication  and  performance  of  this  experimental 
test  facility  has  been  reported  in  the  referenced  document.^ 

The  participants  all  agreed  that  the  cost  of  facilitizing  to 
perform  these  tests  will  be  very  expensive  and  raise  many  questions 
of  cost  evfectiveness . 

^A.  Mayer  and  D.  Prather,  An  Experimental  Facility  for  Dynamic 
Combined  Environments  Reliability  Testing  of  Airborne  Equipment, 
AFFDL-TR- 76-108,  1976. 
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One-third  of  the  15  questions  were  related  to  this  part  of  the 
study.  The  specific  responses  to  these  questions  are  included  in 
the  overall  distribution  of  responses. 

The  perceptions  of  the  participants  in  this  study  are  reflected 
in  their  responses  to  the  following  15  questions.  These  responses 
do  not  represent  completely  the  opinion  of  any  one  DOD  agency  or 
industry,  nor  all  of  the  participants.  It  does  attempt  to  repre- 
sent a summary  of  the  various  perceptions  held  by  the  participants. 

I hope  that  my  conclusions  and  recommendations  will  be  of  value  in 
influencing  the  content  of  the  reliability  document  currently 
being  revised,  to  improve  the  overall  reliability  of  military  wea- 
pon systems,  and  possibly  lead  to  a reduction  in  the  cost  of 
ownership . 

PARTICIPANTS’  RESPONSES  TO  SPECIFIC  QUESTIONS 
OF  THE  STUDY 

The  remainder  of  the  chapter  presents  responses  from  the 
participants  to  the  15  questions  presented. 
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1. 


WHAT  IS  YOUR  PERCEPTION  OF  THE  CURRENT  DOD 
RELIABILITY  TEST  PHILOSOPHY? 


Responses 

Tne  current  philosophy  is  more  directed  toward  compliant  > 
military  standards  and  specifications  rather  than  towards  the 
equipment  reliability. 

One  which  has  over-emphasized  expensive  techniques  for  labora- 
tory testing  with  little  or  no  appreciation  of  tne  enormous 
burden  it  will  place  on  DOD  activities,  or  the  manner  in  which 
the  contractors  can  undermine  it. 

I believe  it  to  be  unnecessary,  expensive,  and  ineffect ive--yet 
one  whicn  ignores  very  real  opportunities  to  achieve  consider- 
aole  improvement  at  relatively  little  cost. 

Can't  speak  for  DOD  overall  pnilosophy . I nave  trouble  tniuking 
above  our  command.  I don't  think  tnere  is  a DOD  reliability 
test  philosophy--and  I don't  think  that's  a problem. 

I don't  know. 

A philosophy  that  depends  almost  exclusively  on  "Test-Find  and 
Fix"  to  control  reliability. 

A philosophy  that  places  too  much  emphasis  on  testing  and  not 
enough  on  design. 

Not  sure. 

The  reliability  test  philosophy  is  as  defined  in  DOD  Directives 
5000.1-3  and  it  is  becoming  more  realistic  and  can  be  tailored 
to  suit  the  acquisition  strategy  on  a case  by  case  basis  . 

One  that  is  improving. 

A philosophy  that  drives  by  cost  and  schedule. 

One  that  fails  to  address  the  many  factors  that  influence  relia- 
bility . 

A philosophy  that  is  effective  for  very  large  or  major  programs 
but  not  effective  on  small  programs. 

No.  But  what  we  need  is  a common  design  philosophy--that ' s the 
key  issue.  Test  plans  or  test  philosophy  won't  solve  the 
problem.  We  need  a type  design  engineer  that  understands  tie 
problems  of  his  technologies  he's  working  with.  Reliability 
should  be  a religion  for  every  design  engineer. 
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2.  DO  YOU  THINK  DOD  HAS  A COMMON  DEFINITION  OF  WHAT  CONSTITUTES 


A RELIABLE  SYSTEM,  i.e.,  FIELD  RELIABILITY  TO  THE 
LOGISTICIAN  MAY  NOT  MEAN  THE  SAME  THING  TO  THE 
DESIGNER  OR  CONTRACT  ADMINISTRATOR? 


Responses 

_ No.  Not  a common  definition. 

_ No.  However  considerable  effort  is  being  made  to  improve  the 
definition  problem. 


We  will  have  a common  definition  if,  and  only  if,  the  definition 
of  the  RAM  parameters  are  understood  by  all. 


Yes.  I think  the  definition  of  a reliable  system  is  generally 
the  same  throughout  DOD,  and  that  is  how  long  does  the  equip- 
ment operate  under  mission  environment? 


No.  No  common  perception  of  field  problems  reliability. 

No.  A considerable  area  of  difficulty  exists  just  because  of 
these  conceptual  differences. 


Yes  . 

No.  I think  we  have  confused  a bunch  of  people. 

At  present,  the  DOD  does  not  have  a common  definition  of  what 
constitutes  a reliable  syst  -m.  The  present  way  that  relia- 
bility is  specified  in  contractual  documents  and  how  the 
equipment  is  demonstrated  are  not  compatible. 


3. 


DOES  THE  NEED  EXIST  FOR  DOD  TO  PROVIDE  MORE 


GUIDANCE  OR  A POLICY  DIRECTIVE  TO  ESTABLISH 
A MORE  REALISTIC  RELIABILITY  PROGRAM? 


Responses 

Yes.  But  it  should  be  guidance--not  a "how-to  (cook  book)."  By 
providing  a DOD  directive,  it  should  give  the  program  the  high 
level  authority  it  needs. 

No.  More  guidance  would  be  more  harmful  than  helpful. 

Yes.  We  need  broad  guidance  with  options. 

Yes.  More  management  emphasis. 

Yes.  Realistic  is  the  key  word.  The  "conventional  wisdom"  of 
what  wrong  with  the  current  program  is  dangerously  misinformed 
for  example,  there  is  virtually  no  real  understanding  of  just 
why  equipment  in  the  field  fail  and  without  such  understanding 
we  can  only  provide  opinions  of  what  should  be  done  to  fix  the 
system  (no  matter  how  widely  subscribed  to)  . 

Yes  . 

Yes.  And  I think  one  is  being  written. 

No.  We  need  better  understanding  of  the  existing  guidance. 

Yes.  The  existing  reliability  documents  should  be  pulled 
together  and  the  proposed  DOD  Directive  should  provide  this 
guidance . 
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WHAT  FACTORS  OTHER  THAN  ENVIRONMENTAL  INFLUENCE 


POOR  RELIABILITY  IN  THE  FIELD? 


Responses 

Integrated  Logistic  Support,  Maintenance  Support,  Operations 
and  Support  Cost. 

System  Interaction  Effect— equipment  which  causes  other 
equipment  to  fail. 

Software  which  produces  failure— or  the  appearance  of  failure. 

Improper  maintenance  which  causes  more  failure  than  natural. 

Equipment  delivered  to  the  field  which  does  not  have  the 
reliability  characteristic  which  should  have  resulted  from 
the  design/test  effort. 

Cost  cutting  schemes  in  the  Procurement  Cycle. 

Inadequate  and  insufficient  field  test  or  early  feedback  from 
field  deployment. 

Tech  orders  that  have  not  been  thoroughly  checked  out . 
Inexperienced  personnel  (operators). 

Interface  problems . 

Failure  definitions. 

Inadequate  reporting  system. 

The  environmental  factors  are  the  major  contributors  to  the 
unreliability  of  equipment.  Other  factors  which  influence  poor 
reliability  are: 

a.  Ease  of  equipment  repair.  Our  equipment  which  is  easily 
disassembled,  repaired  and  checkout  is  less  likely  to  be 
"screwdrivered"  and  damaged. 

b.  Misapplication  of  the  equipment.  If  the  equipment  is 
designed  to  perform  a specific  task,  it  should  not  be  used  for 
other  applications. 
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IN  YOUR  OPINION,  DO  WE  ADEQUATELY  MONITOR  THOSE 
FACTORS  THAT  EFFECT  EQUIPMENT  RELIABILITY? 


Responses 

Lack  of  funds  prevent  participation  and  observation  of  the 
reliability  techniques. 

Lack  of  clear  R&M  definitions  and  difficulties  in  writing 
specifications . 

It  is  difficult  to  enforce  reliability  requirement  under  the 
current  DOD  acquisition  system  because  of  the  heavy  emphasis 
on  cost  schedule  or  aircraft  flight  conditions. 

During  the  production  phase,  yes  it  is  monitored  adequately; 
however,  we  need  a better  system  for  monitoring  the  repair 
and  maintenance  cycle. 

No.  They  don't  seem  to  have  enough  adequately  trained 
personnel  in  the  field  of  reliability. 

No,  not  adequate  job.  The  Air  Force  in  my  opinion  works 
closer  with  the  developer  than  the  other  two  services. 

No.  All  of  the  Service  reliability  monitoring  and  data 
collection  are  poor  at  best.  Great  strides  in  equipment 
reliability  can  be  made  with  proper  documented  failure  data 
from  the  field  and  other  programs.  Once  the  problems  is 
identified,  the  corrective  action  is  easy. 


6. 


CAN  THE  RELIABILITY  OF  AVIONIC  SUB-SYSTEMS  BE 


IMPROVED  BY  IMPROVING  THE  RELIABILITY  TEST 
METHODOLOGY  DURING  THE  EARLY  PHASE  OF 
THE  ACQUISITION  CYCLE? 


Yes.  The  use  of  Reliability  Test  during  the  Engineering 
Development  Phase  of  Avionic  equipment  can  prove  the  ultimate 
reliability  of  the  system. 


Yes.  Front-end  lead  the  RAM  testing  through  design,  more 
dedicated  funding. 


Sure.  It  is  the  most  cost  effective  approach. 


Yes.  However,  the  first  goal  should  be  to  adhere  to  781B  test 
philosophy  because  the  most  serious  problem  in  existing  equip- 
ment can  be  treated  to  abuse  of  the  currently  published  (781B) 
requirements . 

Second  goal  should  be  improvement  of  existing  requirement  and 
documents . 
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DO  WE  DO  AN  ADEQUATE  JOB,  CONTRACTUALLY, 

DEFINING  reliability  requirements? 


They  are  beginning  to  do  a much  better  job  defining  reliability 
requirements  and  I think  the  new  DOD  Directive  5000. X will  help 
them  do  even  a better  job. 


Yes;  they  do  a good  job  defining  Reliability  contractually,  but 
the  real  question  is  how  realistic  is  the  MTBF  number.  I don't 
think  they  do  a good  job  in  establishing  what  that  MTBF  should 
be. 


No.  I think  we  need  to  improve  in  this  area  and  I think  we 
are  improving. 


No.  However,  there  is  a Joint  Technical  Coordination  Group 
on  Electronic  Systems  reliability  working  to  improve  the 
situation. 


No.  Definitely  not. 

I think  the  overall  opinion  is  no,  for  any  type  system. 

No.  There  is  a difference  between  Services,  but  within  the 
Service  we  still  find  the  same  kind  of  variance. 


Yes.  MTBF  requirements  are  defined  in  the  correct  number, 

but  they  are  not  based  on  analysis  of  the  operating  environments. 

No!  The  total  field  is  never  identified  MIL  SPE  call  out  sucli 
as  MIL  STD  810  and  MIL-E  5400,  Class  0 equipment  does  not  define 
all  the  environmental  requirements.  The  government  must 
identify  all  of  the  factors  which  effect  equipment  reliability. 


8.  DOES  THE  CURRENT  LOGISTIC  REPORTING  SYSTEM  PROVIDE  THE  KIND 
OF  DATA,  FROM  THE  FIELD  NECESSARY  TO  IMPROVE  THE 
RELIABILITY  OF  AVIONICS  SUB-SYSTEMS? 


Responses 

No,  Leaves  a lot  to  be  desired. 

No.  Only  through  sample  data  collections,  obtaining  reliability 
data  on  fielded  equipment  is  expensive  and  most  maintenance 
personnel  are  not  trained  to  prove  that  kind  of  data. 

No . 

Definitely  not.  There  are  many  problems  and  issues  in  tiiis 
arena.  The  most  serious  is  the  absence  of  any  good  effort 
to  provide. 

No.  The  type  of  data  now  being  reported  is  fine  for  identifying 
trends  and  for  logistic  support,  but  the  kind  of  data  required 
for  engineering  or  reliability  improvement  does  not  exist. 

No.  Many  of  the  failures  counted  in  the  field  are  caused  by 
maintenance  personnel . 

No.  Obtaining  engineering  data  from  the  field  will  always  be 
difficult,  but  to  not  go  to  the  extra  expense  and  effort  during 
early  deployment  to  gain  insight  into  reliability  and 
engineering  problems  is  to  miss  a valuable  opportunity. 

This  is  currently  a problem  area. 

No.  Data  requirements  should  be  tailored  to  system  complexity 
and  specific  needs. 

No.  Failure  definition  was  identified  as  part  of  the  problem. 

The  reporting  forms  vary  with  each  service  for  a given  system. 
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9. 


DOES  THE  DOD  REPRESENTATIVE  PROVIDE  THE  EQUIPMENT 


■ 


DESIGNER  AND  THE  RELIABILITY  ENGINEER 


ADEQUATE 


INFORMATION  DURING  THE  INITIAL  PHASE  OF 


THE  PROGRAM? 


Responses 

No.  There  is  a definite  need  for  the  DOD  reliability  represen- 
tative to  work  very  close  with  the  developer--specif icallv 
the  designer  and  the  reliability  to  make  all  of  the  informa- 
tion required  of  the  system — Environmental  and  Operational 
requirement.  This  should  include  the  maintenance  required 
and  the  level  of  training  of  the  equipment  operation. 

No. 

Can  only  speak  to  our  situation--not  all  DOD--and  the  answer  is 
no.  But  I don't  consider  it  a major  deficiency. 

No.  This  is  the  program  manager's  job.  Maybe  ne  needs  a 
reliability  handbook  and  some  training  in  reliability  and 
cost  trade-off. 

Yes.  I think  the  requirements  such  as  mission  profile  on  new 
systems  should  be  the  developer's  responsibility  and  we 
should  provide  funds  for  it  through  the  contract. 

No  information  is  required  on  the  total  environment,  including 
the  operating  mode,  specific  location  in  the  aircraft,  the 
capability  of  the  maintenance  support  personnel. 

No.  The  need  exists  for  a reliability  and  maintainability  data 
base  and  this  information  should  be  available  to  all  contractors 
so  that  working  with  the  Services  the  contractors  could  also 
contribute  to  this  data  base. 
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10.  WHAT  IS  YOUR  PERCEPTION  OF  OUR  ABILITY  TO  PROVIDE  THE 

CONTRACTORS  WITH  REALISTIC  MISSION  PROFILES? 

Responses 

The  AF  and  Navy  have  improved  in  this  area.  I have  not  had 

enough  experience  with  the  Army  to  comment  on  the  Program. 

Improving,  but  they  still  have  a long  way  to  go.  For  example, 

the  Air  Force  at  Wright  Patterson  may  have  to  develop  a 
technique  or  compile  a good  data  base  and  the  other  service 
may  have  no  knowledge  of  this  capability. 

_ The  capability  exists.  More  funds  or  emphasis  is  required. 

_ Inadequate  because  it  is  hard  to  obtain  data  from  the  field 
during  early  deployment. 

_ Inadequate  because  of  budget  constraint . 

Fair,  the  new  MIL-STD  781C  should  improve  this  situation. 

_ Not  a very  good  job.  The  MTBF  selected  quite  often  is  not  based 
on  realistic  requirements. 

_ Very  1 imited . 

On  the  F16  Program  we  established  a mission  profile  based  on  our 

knowledge  of  the  aircraft  utilizing  the  assist  of  our  pilot. 
The  Air  Force  personnel  worked  with  us  drawing  on  their 
experience  of  other  systems.  The  capability  exists  to  provide 
the  information,  but  what  is  required  is  an  information 
exchange  program  for  the  existing  data.  ItS  my  observation 
that  very  little  of  the  information  available  is  being  shared. 


11.  HOW  MUCH  RELIABILITY  CONSIDERATION  SHOULD  BE  GIVEN  TO 
THE  CONCEPTUAL  AND  INITIAL  DESIGN  PHASE? 

Responses 

_ As  much  as  possible.  This  is  where  the  planning  and  trade-off 
begin.  You  increase  your  cost  by  not  considering  the 
reliability  requirements  during  the  conceptual  phase. 

_ As  much  as  possible  the  planning  and  testing  is  vital  during 
this  phase  of  the  program. 

_ This  is  where  the  reliability  requirements  should  begin  to  take 
shape.  Attempts  should  be  made  to  review  like  systems  in  the 
field.  The  requirements  and  the  reliability  assessment  methods 
should  be  clearly  defined. 

_ The  conceptual  and  initial  design  phase  is  where  the  basic 

decisions  on  the  approach  to  obtain  the  specific  reliability 
requirements  is  developed.  The  thermal  design  outline  and  part 
selection  approach  is  established. 

Follow  the  guidance  of  MIL-STD  785,  it's  adequate  for  the 

conceptual  phase. 

More  consideration  than  we  are  currently  giving. 


Depends  on  the  type  of  program,  if  it  is  major  or  xmall  box. 

We  are  more  in  favor  of  growth  kind  of  approach,  where  you 
uncover  problem  areas  and  you  either  fix  it  or  identify  the 
fix  and  not  get  involved  in  a bean  counting  game.  A pass 
fail  criteria  should  be  established.  The  test  method  should 
be  one  of  uncovering  weakness,  design  deficiencies,  and 
provide  a procedure  for  resolving  failures. 

During  the  conceptual  phase  we  should  be  establishing  the  design 
requirement  and  test  approach  necessary  to  ensure  that  the 
equipment  will  meet  the  pre-production  reliability  demonstra- 
tion test. 

Quantitative  R&M  requirements  should  be  established,  these  values 
should  be  based  upon  realistic  user  requirements  giving  due 
consideration  to  the  weapon  systems  mission  profile  and  minimum 
essential  operational  needs  in  the  anticipated  environment  to 
include  optimizing  the  life  cycle  cost. 
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HOW  MUCH  CONSIDERATION  SHOULD  BE  GIVEN  TO  THE 


12  . 


RELIABILITY  REQUIREMENTS  DURING 
ENGINEERING  DEVELOPMENT  TESTING? 


Responses 

As  much  as  you  can,  you  start  your  testing  as  soon  as  you  have 
hardware,  burn  in  at  the  earliest  phase  possible,  many  design 
problems  are  possible. 

Both  the  contractor  and  the  Government  should  be  working  as  a 
team  in  planning  reliability  and  assessment  techniques  to 
allow  for  anticipated  achieved  reliability. 

The  reliability  consideration  should  have  started  during  the 
conceptual  phase; at  this  phase  of  the  program  you  should  be 
conditioning  those  tests  that  will  provide  data  for  timely 
tow-risk  full-scale  development  of  the  new  system.  For 
example,  Breadboard  model  mock-up  test,  engineering  developed 
hardware  test,  prototype  hardware  test,  early  production 
hardware  as  a continuous  process. 

More  than  is  currently  given  to  reliability  consideration. 

As  much  as  possible  both  the  reliability  requirement  and  testing 
during  this  phase  is  vital  to  success. 

No  consideration,  or  practically  no.  It  could  be  counter  produc- 
tive for  many  (but  not  all)  programs  assuming  the  goal  is  to 
find  and  fix  as  many  engineering  problems  as  possible  during 
this  phase  and  not  to  count  failure  or  be  concerned  with 
correlation  of  laboratory  vs.  field  data. 

The  emphasis  on  reliability  should  be  applied  most  heavily  during 
Engineering  development  tests.  This  is  the  phase  where  the 
most  cost  effective  corrections  can  be  accomplished.  Failure 
modes  can  be  identified  and  corrective  actions  developed, 
design  changes  can  be  made. 

Reliability  on  the  total  RAM  requirement  should  be  the  most 
critical  issue  in  all  DCPs . That's  the  only  way  it  ever  will 
succeed . 


13. 


WHAT  SHOULD  BE 
ENGINEERING 


THE  ROLE  OF  THE  GOVERNMENT  DURING  THE 
DEVELOPMENT  TEST,  i.e.,  TEST  FIND 
AND  FIX? 


Responses 

Deeply  Involved  with  the  contractor  in  a role  of  assisting  in 

the  process  of  identifying  and  finding  solutions  to  failure;  not 
one  of  counting  the  number  of  failures. 

The  DOD  involvement  can  be  very  helpful  in  that  it  encourages 
managers  to  provide  support  in  seeking  solutions  to  problems. 
Industry  and  the  DOD  representative  should  work  as  a team. 

A member  of  the  team.  Both  representatives  must  be  qualified 
and  reasonable  people.  If  the  DOD  representative  can  only 
quote  regulations  and  count  failures,  he  can  cause  serious 
cost  and  schedule  impact. 

HELP  find  and  FIX  in  the  role  of  a member  of  the  team. 

Establish  guidance,  monitor  development  and  make  recommendations 
for  improved  reliability. 

Provide  the  incentive  for  the  contractor  to  find  and  fix  as  many 
failures  as  possible. 

Work  more  closely  with  the  developers  during  this  phase  and 
assure  that  adequate  funds  are  available  to  correct  problem 
identified . 

When  it  becomes  necessary,  be  sure  top  management  allows 
adequate  time  to  take  corrective  action,  and  management 
emphasis . 
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WHAT  ARE  YOUR  PERCEPTIONS  OF  THE  COST  IMPACT  OF 


IMPLEMENTING  THE  C VERSION  OF  MIL-STD  781? 


Responses 

Increase  in  test  cost  but  should  produce  a more  reliable  system. 

Increase  facility  and  test  cost.  May  encourage  program  managers 
to  eliminate  some  of  the  test  because  of  the  increased  cost. 

The  cost  impact  will  be  astronomical.  The  main  reason  for  the 
increase  will  be  new  facilities,  test  equipment  and  mainten- 
ance . 

Cost  of  testing  will  be  too  high  (see  Naval  Avionics  Facility 
Report) . 

The  test  cost  will  go  up  and  the  first  programs  will  have  to  pay 
for  the  increased  facility  cost.  See  special  cost  summary 
put  together  by  the  Institute  of  Environmental  Sciences. 

The  long-range  overall  cost  will  be  less.  The  initial  test  cost 
will  be  increased. 

The  use  of  mission/environmental  profile  to  determine  the  equip- 
ment specifications  should  lead  to  more  realistic  test  and 
result  in  an  overall  cost  of  ownership. 

We  don't  know.  At  present  we  are  conducting  the  CERT  evaluation 
program  here  at  Wright-Patterson.  Part  of  the  objective  is  to 
assess  technical  merit  of  combined  reliability  testing  and 
evaluate  the  cost  effectiveness  of  this  type  of  approach.  We 
are  taking  a good  look  at  cost,  to  include  ways  of  reducing 
facility  cost.  Upon  completion  of  test  tasks,  we  will  provide 
implementation  instructions  and  plans. 

I have  seen  a cost  of  studies  and  I think  it  is  too  early  to  know 
what  the  real  cost  impact  will  be.  The  initial  test  cost  and 
the  additional  facilities  will  increase  the  cost  considerably. 

The  Air  Force  should  be  better  qualified  to  answer  the  cost  impact 
because  they  are  currently  making  comparisons  of  the  cost  by  con- 
ducting CERT  test  approach  relative  to  the  current  reliability  and 
environmental  test  of  avionics  equipment.  Capital  investment  is 
being  considered  in  this  evaluation. 

The  Institute  of  Environmental  Sciences  made  a nationwide  study  of 
the  cost  impact  which  reflected  the  increase  facility  cost.  I 
have  forgotten  the  exact  amount,  but  it  will  be  high. 


IF  WE  IMPLEMENT  THE  C VERSION  OF  MIL  SPECIFICATION  781, 
WILL  WE  HAVE  CLOSER  CORRELATIONS  BETWEEN  LAB  TEST 
AND  FIELD  RELIABILITY  DATA? 


Responses 

That  is  a big  concern  of  mine  that  some  people  in  DOD  think 
we  are  to  go  out  and  do  this,  combine  environmental  requirement 
of  781C  and  end  up  with  MTBF  in  the  test  program  that  we  see 
in  the  field.  I do  not  think  that  will  be  true.  I think  that 
three  or  four  years  from  now,  when  we  start  getting  the  781C 
result,  we  will  ask  what  went  wrong.  I do  think  we  will  be  in 
better  shape.  But  it  will  not  correct  the  problem  COL  Swett 
originally  uncovered  in  his  Phase  I Study. 

The  disparity  between  field  reliability  data  is  due  to  a 
great  number  of  things,  some  of  which  are  environmental. 

Yes.  Because  of  increased  test  realism. 

No.  There  are  too  many  variables  not  covered  by  the  781C. 

No.  I think  there  is  an  over  emphasis  on  testing  and  not 
enough  on  design. 

Yes,  there  is  no  doubt  about  the  c version  going  to  do  a better 
job.  The  integrated  test  approach  and  the  more  realistic  test 
approach  should  provide  closer  correlation  of  laboratory 
reliability  and  field  reliability  data. 

The  use  of  mission/environmental  profiles  to  determine  the 
equipment  should  result  in  a much  more  realistic  test  and 
closer  correlation  of  field  and  lab  test  results. 

The  Reliability,  Qualification  and  Acceptance  Tests  are  carried 
out  under  the  combined  environmental  conditions  of  electrical 
input,  temperature,  vibration,  and  humidity  with  the  specific 
test  level  to  be  derived  from  the  miss  ion  and  environmental  pro- 
file for  the  specific  equipment.  This  will  result  in  closer 
correlation . 

We  hope  so,  but  we  are  currently  conducting  the  CERT  evaluation 
program  to  determine  the  technical  merit  of  the  type  of  lab  test 
as  outlined  in  781C.  When  these  tests  are  complete,  we  will 
compare  the  results.  At  present  I would  rather  not  predict  the 
outcome . 

Yes,  but  not  much  closer  because  of  the  logistic  reporting  system. 
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CHAPTER  V 


SUMMARY,  CONCLUSIONS,  AND  RECOMMENDATIONS 

The  role  of  Reliability  and  Environmental  Engineers  in  the  past 
has  been  traditionally  a limited,  supportive  one.  This  assumed  role 
was  accepted  by  top  management  and  decision-makers  in  the  military 
and  industry.  Until  recently,  very  little  effort  was  made  to 
integrate  their  expertise  early  in  the  acquisition  process  even 
though  many  of  the  decisions  required  their  technical  expertise.  It 
was  not  until  DOD  established  an  intensive  program  to  understand  and 
follow-up  on  the  means  to  significantly  improve  field  reliability  of 
complex  weapons  system  that  we  began  to  see  changes  in  these 
practices  . 

These  changes  are  coming  about  as  a result  of  such  issues  as 
the  need  to  reduce  support  cost,  increase  equipment  readiness,  and 
improve  overall  reliability  of  new  weapons  systems. 

This  study  identified  some  of  the  more  significant  issues  and 
problems  associated  with  these  changes  in  achieving  improved 
reliability  for  avionics  systems  and  subsystems. 

SUMMARY 


The  purpose  of  this  study  was  to  explore  first  some  of  the 
issues  and  problems  associated  with  reliability  for  avionics 
systems  and  subsystems;  and  second,  to  examine  some  of  the  implica- 
tions of  the  c version  of  MIL-STD  781;  also,  to  explore  the  relia- 
bility problems  associated  with  Engineering  Development 
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(test-analyze-find  and  correct)  testing  of  avionics  subsystems  and 
equipment.  Additional  research  was  conducted  to  provide  recommenda- 
tions that  may  result  in  improved  engineering  development  test 
approaches  and  contribute  to  reduction  in  the  cost  of  ownership. 

The  study  considered  the  broad  principles  recommended  by 
earlier  investigations  and  attempted  to  obtain  specific  data  which 
would  lead  to  approaches  that  will  enhance  the  existing  reliability 
test  approach  during  Engineering  Development  testing,  and  possibly 
lead  to  a reduction  in  operation  and  support  cost  for  avionic 
equipment . 

Problem  areas  have  been  identified  and  an  attempt  made  to 
assess  the  magnitude  of  the  problems  and  to  determine  their  prin- 
cipal causes. 

In  addition  to  the  series  of  previous  studies  conducted  on 
avionics  equipment  reliability,  the  Joint  Logistics  Commanders  and 
the  Joint  Technical  Coordinating  Group  for  Electronics  Systems 
Reliability  has  recently  established  the  following  task  groups: 

(1)  Reliability  Engineering  data  requirements  and  collection,  (2) 
Reliability  test  and  analysis,  (3)  Reliability  acquisition  manage- 
ment, (4)  Software  Reliability,  (5)  Reliability  Design,  and  (6) 
Reliability  Training.  These  groups  are  addressing  many  factors 
which  contribute  to  low  operational  reliability.  These  factors 
include  steadily  increasing  system  complexity,  poor  management 
practices,  inadequate  documentation,  unrealistic  test  requirements 
and  procedures,  poor  design  practice,  unrealistic  environmental 
requirements,  and  software  errors.  These  factors  are  in  concurrence 

with  the  findings  of  this  study. 
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CONCLUSIONS 


The  conclusions  reached  suggest  a number  of  areas  for  improve- 
ments. I have  had  the  rewarding  task  of  reconciling  the  well- 
considered  and  differing  perception  of  the  participant  responses. 
These  conclusions  do  not  represent  completely  the  opinions  of  any 
one  agency  nor  all  of  the  opinions  of  the  participants.  The  conclu- 
sions and  recommendations  represent  the  viewpoints  of  the  researcher 
which  takes  into  consideration  inputs  from  all  of  the  participants. 

The  conclusions  of  the  study  are: 

1.  There  was  evidence  that  no  standard  methodology  exists  for 
developing  comprehensive  reliability  test  environments.  This  lack 
of  methodology  is  reflected  in  inadequate  reliability  test  environ- 
ments which  result  in  a set  of  performance  and  test  criteria  which  is 
not  representative  of  the  true  mission  requirements  of  the  equipment. 

2.  There  was  evidence  that  the  reliability  requirements  and 
acquisition  decisions  related  to  avionics  systems  and  subsystems 
are  defined  in  terms  of  performance,  physical  characteristics,  cost, 
quantity  and  schedule  in  conformity  with  the  perceived  threat  and 
need.  The  overall  requirements  of  the  acquisition  and  decision 
process  includes  attention  to  all  of  these  components;  however,  the 
current  approach  appears  to  be  driven  by  cost  and  schedule  only, 
with  inadequate  consideration  given  to  reliability  in  the  early 
phase  of  the  acquisition  and  decision  process. 

3.  That  inconsistencies  exist  between  the  services'  mainten- 
ance reporting  system  (Navy,  3-M;  Air  Force,  66-1;  and  the  Army, 
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TAMMS)  for  calculations  of  reliability.  At  present  each  of  the 
three  services  has  its  own  reliability  standards  (see  Appendixes 
B,  C,  D).  The  typical  service  maintenance  reporting  system  opera- 
ting time  is  used  as  the  basis  for  calculating  equipment  failure 
performance.  This  is  considered  a significant  problem  area. 

4.  There  is  evidence  that  the  need  exists  for  more  failure 
analysis,  time  and  funds  for  engineering  development  testing  of 
avionics  equipment . 

5.  The  need  exists  for  top  management  in  industry  and  DOD  to 
continue  to  transmit  their  concern  for  improved  reliability  and 
provide  proper  managerial  emphasis  throughout  the  development 
process . 

6.  There  is  evidence  that  the  existing  test  program  MIL-STD 
781B  needs  some  restructuring.  The  current  environmental  require- 
ments are  not  realistic  for  most  avionics  equipments. 

7.  The  need  exists  for  a comprehensive  analysis  and  definitive 
program  to  be  carried  out  on  hardware  and  software  for  new  avionics 
during  the  conceptual  and  validation  phase,  prior  to  full-scale 
development.  At  present  inadequate  considerations  are  given  to 
reliability  requirements  for  avionics  system  during  the  conceptual 
design  and  planning  phases. 

8.  There  is  some  evidence  that  the  need  exists  to  formulate 

a procedure  which  will  define  what  a reliability  test  should  include 
during  the  Engineering  development,  and  where  environmental  data 
resides;  how  to  formulate  a procedure  and  how  it  should  be 
incorporated  into  a contract. 
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9.  That  the  reliability  of  avionic  subsystems  and  systems  can 
be  improved  by  improving  the  reliability  test  methodology  during  the 
early  phase  of  the  acquisition  process. 

10.  The  need  exists  to  perform  more  reliability  analysis,  to 
include  failure  rates,  cause  of  failure,  failure  mode  and  effect 
analyses,  and  to  review  subsystem  allocations  to  the  equipment  level 
and  verify  feasibility  of  the  allocated  requirements. 

11.  The  contractors  are  not  being  provided  the  information 
they  need  to  design  equipment  for  reliable  performance  in  field 
service. 

12.  More  emphasis  should  be  placed  on  reliability  growth 
during  Engineering  development . 

13.  There  is  a marked  absence  of  DOD  management  policy  guidance 
regarding  a comprehensive  approach  to  reliability  testing  for 
avionic  equipment . 

The  findings  outlined  in  Chapter  IV  provide  some  means  of 
understanding  the  nature  of  the  problems  associated  with  avionics 
reliability.  The  task  of  integrating  performance,  reliability,  and 
environmental  considerations  into  a rational  approach  that  complies 
with  existing  DOD  policy  is  an  acute  challenge. 

RECOMMENDATIONS 

Following  are  recommendations  for  making  more  effective  use  of 
existing  test  time  and  possibly  improving  avionics  equipment  field 
reliability.  These  recommendations  are  based  on  data  obtained  from 
the  study. 
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1.  DOD  provide  guidance  that  will  result  in  a standard  metho- 
dology for  developing  comprehensive  reliability  test  environments. 
This  can  be  accomplished  by  ensuring  that  the  military  departments 
are  responsible  for: 

a.  Establishing  realistic  environmental  requirements.  The 
requirements  should  be  based  on  mission/environment  profiles  that 
are  derived  from  the  life  cycle  profile  as  defined  by  the  stated 
operational  requirements  for  the  avionics  systems  or  subsystems. 

b.  Ensure  that  there  is  adequate  coordination  between  the 
operating,  developing,  and  supporting  commands. 

c.  Provide  contractors  with  all  information  that  influences 
test  criteria  and  performance  as  early  as  possible. 

2.  More  emphasis  be  placed  on  reliability  requirements  during 
the  conceptual  phase  or  early  in  the  acquisition  process.  This  can 
be  accomplished  by  defining  the  reliability  requirements  as  early 
as  possible  and  have  industry  to  comment  cn  draft  RFP  before  a 
competitive  quota  is  prepared. 

3.  DOD  evaluate  the  current  logistic  reporting  system  as  it 
relates  to  reliability.  These  two  areas  are  so  different  it  would 
be  appropriate  to  investigate  the  specifics  of  each  area  in  detail 
and  develop  the  methodology  and  criteria  that  would  provide  an 
independent  assessment  of  mission  and  logistic  MTBF  performance. 
Failure  definitions  and  failures  that  occur  during  nonoperating  time 
should  also  be  investigated. 
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4.  Adequate  time  and  funds  be  made  available  in  the  early  part 
of  the  program  to  test-find  and  correct  deficiencies;  and  that  more 
time  be  made  available  for  failure  analysis  to  determine  the  causes 
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of  the  failures. 

5.  The  management  environment  be  reevaluated  to  ensure  that 
proper  emphasis  is  placed  on  reliability.  The  management  environment 
be  streamlined  so  that  the  program  manager  can  run  the  programs 
without  requiring  decisions  from  several  people  who  are  not  account- 
able or  responsible  for  the  program,  similar  to  the  management 
environment  in  private  industry. 

6.  Environmental  requirements  for  avionics  systems,  including 
subsystems,  be  based  only  on  mission  profiles.  This  profile  should 
take  into  account  the  life  cycle  profile  as  defined  in  the  opera- 
tional requirement  for  the  equipment. 

7.  Identify  each  location  of  each  s-bsystem  of  avionics  equip- 
ment in  the  deployed  aircraft,  i.e..  the  wing  vs.  the  cockpit. 

8.  Obtain  flight  profiles  data  from  other  aircraft  carrying 
similar  equipment.  Many  of  the  subsystems  will  fly  on  different 
aircraft  and  have  multi-missions. 

9.  Mere  emphasis  and  detail  planning  for  reliability  to  be 
accomplished  during  the  conceptual  phase.  Software  and  hardware 
testing  and  requirement  definitions  should  also  be  given  more 
consideration. 

10.  The  military  departments  should  provide  adequate  guidance 
so  the  contractors  understand  what  tests  should  be  included  during 
the  development  phase.  A procedure  should  be  developed  that  assures 
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adequate  planning  prior  to  starting  the  testing,  and  provides  a 
document  that  reflects  an  understanding  of  the  agreement  between 
the  government  and  the  contractor  specifying  the  intent  of  the  test. 
The  criteria  for  failure  and  the  procedural  rules  should  be  included. 

11.  The  reliability  test  starts  as  early  as  possible  in  the 
acquisition  process.  Adequate  funds  to  be  made  available  so  that 
most  failures  can  be  found  and  corrected  during  the  engineering 
development  phase.  The  levels  of  tolerance  for  time,  environmental 
stress  levels,  including  appropriate  combined  stress,  and  environ- 
mental cycle  Co  be  conducted  during  engineering  development  testing. 

12.  Establish  a comprehensive  reliability  data  base  of  previous 
analyses,  caused  failures,  and  reevaluate  the  current  approach  to 
allocating  subsystems  requirements. 

13.  Provide  contractors  with  information  they  desire  to 
design  operational  reliability  into  the  systems.  This  will  require 
more  government  involvement  and  also  more  coordination  between  the 
services . 


recommendations  of  the  researcher 


These  additional  recommendations  are  based  on  the  personal 
experience  of  the  researcher  in  the  field  of  reliability  testing. 

Reliability  growth  be  given  more  consideration  during  the 
Engineering  Development  Test,  especially  during  the  test-analyze- 
find  and  correct  phase.  This  will  include  screening  and  replacing 
failure  parts  and  the  reliability  caused  by  incorporation  of 
appropriate  design  changes.  This  type  of  consideration  prevents 
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the  discovery  of  problems  during  the  prototypes  from  being 
transferred  to  the  production  units. 

To  provide  guidance  to  each  of  the  military  departments,  a 
DOD  reliability  policy  directive  be  established. 

The  contracting  approach  provide  incentive  for  the  contractor 
to  meet  or  exceed  the  reliability  requirement. 

The  attitude  or  belief  that  reliability  can  be  achieved  at 
the  end  of  a development  program  and  can  be  demonstrated  should  be 
changed.  Reliability  is  a function  of  many  things  that  happen  after 
the  design,  most  of  these  have  been  discussed  in  the  study. 
Therefore,  reliability  should  be  viewed  as  a continuous  process 
through  operational  deployment  and  the  life  of  the  program. 

During  the  design  phase,  both  parties  should  be  qualified  and 
willing  to  participate  jointly  and  very  candidly  in  identifying 
the  operating  environment  and  other  factors  that  will  affect 
reliability  and  life  cycle  cost. 

Utilize  a planned  design  update;  it  offers  potential  life  cycle 
reduction  through  reliability  and  other  improvements,  increases 
functional  capability,  and  extends  the  useful  lifetime. 

Findings  and  recommendations  of  previous  studies  conducted 
within  the  past  three  years  be  summarized  and  investigated  for 
possible  implementation. 

The  acquisition  cycle  for  avionics  equipment  be  shortened  and 
separated  from  the  long  development  cycle  for  the  aircraft  because 
of  the  changing  technology  of  avionics  equipment. 

Avionic  systems  and  subsystems  be  integrated  and  tested  as  an 
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integral  system  in  the  operational  environment  prior  to  completion 
of  Engineering  development  tests. 

Automatic  test  equipment  and  the  associated  software  be  given 
the  same  type  of  reliability  consideration  that  we  proposed  for  the 
hardware . 

Reduce  the  overall  test  time  by  integrating  the  tests  where  pos- 
sible. There  should  be  more  coordination  and  a team  approach  between 
government  and  industry.  The  development  test  and  the  operational 
test  should  be  integrated  to  eliminate  duplication  of  tests. 

Establish  a budget  for  engineer  development  testing  during  rhe 
time  plans  are  being  formulated  for  avionics  systems  and  subsystems. 

AREAS  FOR  FURTHER  INVESTIGATION 

1.  As  discussed  in  the  conclusions  and  recommendations  formed 

as  a result  of  this  study,  a number  of  areas  appear  worthy  of 
further  investigation.  One  such  area  is:  What  is  the  approach  for 

incorporating  the  logistic  requirements,  human  factors  reliability 
and  operating  conditions  factors  into  the  design? 

2.  Another  one  is  the  assessment  of  reliability  as  it  relates 
to  inherent  MTBF  and  logistic  support,  Logistic  KTBF.  These  are 
two  separate  areas,  and  it  would  be  appropriate  to  investigate  the 
specifics  of  each. 

3.  Investigate  to  determine  what  potential  savings  can  be 
realized  by  shortening  the  development  cycle  of  avionics  equipment. 

A part  of  this  investigation  should  be  an  assessment  of  the  manager 
structure,  i.e.,  the  number  of  approvals  required  for  decisions. 
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AF  REGULATION  80-5 

RELIABILITY  AND  MAINTAINABILITY  PROGRAMS  FOR 
SYSTEMS,  SUBSYSTEMS,  EQUIPMENT, 

AND  MUNITIONS 


AF  REGULATION  80-5 


DEPARTMENT  OF  THE  AIR  FORCE 
Headquarters  US  Air  Force 
Washington  DC  20220 


2 July  1972 


Research  and  Development 

RELIABILITY  AND  MAINTAINABILITY  PROGRAMS 
FOR  SYSTEMS,  SUBSYSTEMS,  EQUIPMENT,  AND  MUNITIONS 


This  regulation  outlines  the  objective,  basic  concepts,  and  policy  of  the  Air  Force  Reliability  and 
Maintainability  (R&M)  Program  for  systems,  subsystems,  equipment,  and  munitions,  and  assigns 
responsibility  for  developing  and  managing  them.  These  policies  apply  to  both  development  and 
operational  hardware;  for  more  specific  guidance  on  operational  equipment,  see  AFR  400-46. 
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SECTION  A — TERMS,  OBJECTIVES, 
AND  BASIC  CONCEPTS 

1.  Terms  Explained: 

a.  Exploratory  and  Advanced  Develop- 
ment. See  AFR  80-1. 
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see  signature  page.) 
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b.  Communications  — Electronics  Imple- 
mentation Plan  (CEIP).  See  AFR  100-2. 

c.  Development  Concept  Paper  (I)CP).  See 

AFR  800-2. 

d.  Hardware  Reliability.  The  probability 
that  a part,  component,  subassembly, 
assembly,  subsystem,  or  system  will  perform 
for  a specified  interval  under  stated  conditions 
with  no  malfunction  or  degradation  that 
requires  corrective  maintenance  actions.  Hard- 
ware reliability  may  also  be  expressed  in 
terms  such  as  Mean  Time  Between  Failure 
(MTBF)  or  Mean  Time  Between  Maintenance 
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Actions.  (Time  should  be  expressed  as  operat- 
ing time  unless  otherwise  specified.) 

0.  Operational  Reliability.  The  probability 
that  an  operationally  ready  system  will  per- 
form as  required  to  accomplish  its  intended 
mission  or  function  as  planned. 

f.  Maintainability.  A characteristic  of 
design  and  installation  expressed  as  the  proba- 
bility that  an  item  will  be  restored  to  a 
specified  condition  within  a given  period  of 
time  when  the  maintenance  is  performed  using- 
prescribed  procedures  and  resources.  Hard- 
ware maintainability  may  also  be  expressed  in 
such  terms  as  Mean  Time  to  Repair,  Mainte- 
nance Man-hours  per  Flying  Hour  or  Mean 
Down  Time. 

g.  Program  Advocacy.  See  AFR  800-2. 

h.  Program  Management  Directive  (PMD). 

See  AFR  800-2. 

1.  Program  Management  Plan  (PMP).  See 

AFR  800-2. 

j.  Required  Operational  Capability  (ROC). 

See  AFR  57-1. 

k.  Source  Selection  Evaluation  Hoard 
(SSEB).  See  AFR  70-15. 

l.  Single  Thread  Data  System  (STDS).  A 

combination  of  data  systems  which  provide  for 
commonality  of  data  elements  and  the  resultant 
capability  to  assess  R&M  performance  of  a sys- 
tem or  item  considering  the  total  data  base 
generated  throughout  its  life  cycle. 

m.  System.  A composite  of  equipment, 
skills,  and  techniques  capable  of  performing  or 
supporting  an  operational  role.  A complete  sys- 
tem includes  related  facilities,  equipment, 
material,  services,  and  personnel  required  for 
its  operation  so  it  can  be  considered  a self- 
sufficient  unit  in  its  intended  operational  and 
support  environment.  (When  the  word 
“system”  is  used  in  this  regulation,  subsystem, 
equipment,  munitions,  and  related  items  also 
apply.) 

2.  Program  Objectives.  The  objectives  of  the 
Air  Force  R&M  prograrp  are  to  contribute  to 
the  performance  and  cost  effectiveness  of  Air 
Force  hardware  by  developing: 

a.  Realistic  R&M  requirements; 


b.  Adequate  provisions  to  demonstrate 
achievement  of  these  requirements  prior  to 
production  or  modification  decisions; 

c.  Management  techniques  to  assure  achieve- 
ment of  R&M  requirements;  and 

d.  An  efficient  basis  for  applying  and  track- 
ing R&M  achievements  for  use  in  both 
engineering  and  logistic  decisions. 

3.  Basic  Concepts: 

a.  Systems  effectiveness  and  cost-effec- 
tiveness are  prime  requisites  in  providing 
Air  Force  systems  for  national  security.  Sys- 
tems effectiveness  is  achieved  through  the 
unified  efforts  of  systems  engineering,  logistics 
and  management  disciplines  in  obtaining  the 
required  performance  requirements.  Cost- 
effectiveness  is  achieved  through  the  trade-off 
of  acquisition  and  support  costs  to  obtain 
desired  system  effectiveness  at  the  lowest  total 
cost  of  ownership. 

b.  Because  reliability  and  maintainability  are 
major  factors  in  system  effectiveness,  in  logis- 
tics support  requirements,  and  in  total  cost, 
throughout  each  system's  life  cycle,  they  must 
be  included  in  each  study  of  system  tradeoffs. 

c.  Reliability  and  maintainability  (R&M)  are 
important  design  and  operational  planning  fac- 
tors; their  achievement  in  the  development,  pro- 
duction and  operation  of  a system  requires  the 
use  of  realistic  requirements.  These  require- 
ments shall  be  imposed  and  enforced  contractu- 
ally by  management  during  the  development 
phase  and  follow-on  phases  of  a weapon  system. 


SECTION  B— AIR  FORCE  POLICY 

4.  R&M  Parameters.  R&M  parameters  shall 
be  identified  as  system  design  requirements 
and  used  during  each  phase  of  the  life  cycle. 
Realistic  and  meaningful  R&M  characteristics 
and  levels  will  be  determined  by  cost  effective- 
ness analysis,  reflecting  both  system  effective- 
ness and  life  cycle  costs.  The  establishment  of 
minimum  acceptable  R&M  levels  must  be 
determined  on  the  basis  of  realistic  operational 
needs. 

a.  Each  Air  Force  organization  that  partici- 
pates in  the  requirements,  development  and 
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acquisition  processes  shall  consult  with  opera- 
tional and  supporting  commands  in  determining 
realistic  R&M"  characteristics  and  objectives 
during  the  conceptual  phase,  using  the  results 
of  studies,  and  prior  experience.  These  charac- 
teristics and  objectives  shall  be  continually  ver- 
ified and  progressively  quantified  throughout 
subsequent  phases. 

(1)  Each  activity  that  prepares  a ROC  will 
consider  prior  experience  with  similar  equip- 
ment, operational  need,  discussions  with  AFSC 
and  AFLC,  and  operational  and  support  con- 
cepts, in  developing  realistic  R&M  design  and 
support  objectives.  It  is  especially  important  in 
this  process  to  describe  the  critical  mission  and 
support  needs  clearly  enough  that  realistic 
R&M  parameters  can  be  derived  from  this 
description.  These  parameters  can  be 
addressed  in  qualitative  or  quantitative  terms, 
depending  upon  the  available  experience.  In 
any  event,  the  ROC  should  describe  the  R&M 
or  critical  mission  and  support  parameters 
clearly,  since  they  serve  as  the  baseline  for 
future  development  and  optimization  efforts. 

(2)  Each  program  document  (such  as  the 
PMD,  CEIP,  and  PMP)  shall  include  realistic 
R&M  design  and  support  objectives,  numerical 
requirements,  and  mission  requirements,  as 
the  status  of  the  system  or  equipment  defini- 
tion allows.  If  these  parameters  cannot  be 
included,  the  document  shall  provide  planning 
guidance  for  the  use  of  the  implementing  com- 
mand in  establishing  these  requirements. 

(3)  The  documentation  prepared  for  pro- 
gram advocacy  during  the  conceptual  phase  shall 
present  the  potential  R&M  levels  involved  in 
each  candidate  system  design  and  describe  any 
special  efforts  or  risks  involved  in  achieving 
them  in  the  development  process. 

b.  Planning  Information.  The  preliminary 
maintenance  concept  and  constraints  on  the 
operation  and  maintenance  of  the  system  will 
he  included  in  each  planning  document  (study, 
Request  for  Proposal,  etc.)  in  sufficient  detail 
to  permit  realistic  modeling  and  to  guide  the 
designer  toward  an  effective  design.  Informa- 
tion to  be  included  in  the  maintenance  concept 
can  be  found  in  attachment  1 of  AFR  66-14. 
Types  of  information  required  will  vary  with 
the  phase  of  development. 

c.  Operational  and  Support  Activities. 

These  activities  shall: 
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(1)  Provide  and  update  R&M  planning 
information  during  the  system’s  conceptual, 
validation,  full  scale  development  and  produc- 
tion phases. 

(2)  Exercise  positive  controls  to  achieve, 
maintain  and  improve  operational  R&M 
throughout  the  life  cycle  of  each  system.  Con- 
duct, under  AFR  66-30  and  4(XM6,  programs 
to  identify  requirements  and  implement  efforts 
to  improve  the  R&M  of  operating  equipment. 

5.  Contract  Requirements.  R&M  program 
plans  of  potential  contractors  will  be  significant 
factors  in  source  selection  actions,  especially  as 
they  impact  system  and  cost  effectiveness.  The 
proposer’s  past  efforts  and  achievement,  his 
proposed  objectives  and  programs,  will  be  con- 
sidered when  sources  are  being  selected.  Con- 
tract incentives  in  terms  of  performance 
guarantees  for  reliability,  maintainability,  sys- 
tem effectiveness,  or  life  cycle  costs,  will  be 
used  when  the  minimum  acceptable  require- 
ments and  demonstration  plans  are  sufficiently 
defined  to  permit  a clear  measurement  of  con- 
tractor performance  (1)01)  Incentive  Contract- 
ing Guide,  AFP  70-1-5).  While  most  Air  Force 
equipments  are  developed  through  contractual 
efforts,  there  is  considerable  work  done  in- 
house  by  the  Air  Force  for  which  R&M  pro- 
grams could  be  appropriate.  Consequently,  this 
regulation  applies  equally  to  in-house  pro- 
grams, projects,  tasks,  or  work  units,  as  well  as 
to  contractual  efforts. 

a.  When  a developmental  effort  is  not 
directly  related  to  an  advocacy  program,  or  to 
the  production  of  specific  hardware: 

(1)  Contracts  for  research  will  be  exempt 
from  R&M  requirements  of  this  regulation. 

(2)  Contracts  for  exploratory  development 
that  do  not  involve  hardware  development,  or 
for  hardware  that  is  to  be  a test  item  produced 
solely  to  demonstrate  the  feasibility  of  a con- 
cept, will  be  exempt  from  the  numerical  R&M 
requirements  of  this  regulation. 

(8)  Contracts  for  Advanced  Development 
of  hardware  shall: 

(a)  Include  numerical  hardware  R&M 
design  objectives. 

(b)  Require  an  analysis  of  all  failures,  to 
determine  the  failure  mode  and  the  probable 
cause,  with  feedback  to  design  activities. 
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(c)  Require  an  update  of  R&M  predic- 
tions, based  upon  engineering  analysis  and  data 
obtained  from  test  programs. 

(4)  Contracts  for  prototyping  shall  not 
ordinarily  require  formal  R&M  programs,  but 
shall  follow  the  policies  outlined  in  subpara- 
graphs (2)  and  (3)  above.  However,  if  the  pro- 
totyping effort  is  planned  to  lead  directly  into 
production,  without  a full  scale  development 
effort,  the  prototyping  contract  will  establish 
numerical  R&M  requirements  which  must  be 
demonstrated.  In  that  case,  applicable  portions 
of  this  regulation  and  MIL  STDs  785  and  470 
will  be  followed. 

b.  When  the  developmental  effort  is  directly 
related  to  program  advocacy  and  obtaining 
approval  of  an  acquisition  program,  each  phase 
described  below  shall  contain  a requirement  for 
contractor  R&M  programs  extending  through 
the  subcontractor  levels. 

(1)  During  the  conceptual  phase,  R&M 
characteristics  and  operational  or  support  con- 
straints shall  be  considered  in  trade-off  studies 
and  quantiative  R&M  objectives  shall  be  estab- 
lished. These  objectives  may  be  modified  in  lat- 
ter phases,  but  must  be  established  during  the 
conceptual  phase. 

(2)  During  the  validation  phase,  quantita- 
tive R&M  values  shall  be  specified.  These 
values  may  be  specified  as  design  objectives 
when  preceding  studies  and  operational  con- 
straints cannot  adequately  define  minimum 
acceptable  and  technically  achievable  values. 
These  values  shall  be  refined  and  firm  require- 
ments established  before  full-scale  development 
phase  begins.  Demonstration  of  these  values  is 
not  mandatory  during  the  validation  phase; 
however,  R&M  predictions,  analysis  of  R&M 
data  from  all  testing,  and  determination  of 
R&M  deficiencies  and  necessary  corrective 
action  shall  be  performed,  documented  and  car- 
ried forward  to  the  full-scale  development 
phase  for  update  and  corrective  action.  A relia- 
bility evaluation  test  (that  is,  a test  to  deter- 
mine reliability  deficiencies  rather  than  to 
demonstrate  achievement  of  specified  values) 
may  be  required  and  be  advantageous  to  Air 
Force  in  the  validation  phase. 

(3)  In  the  full-scale  development  phase, 
the  contract  documentation  shall  include  com- 
prehensive R&M  programs  in  accordance  with 


MIL-STDs  785  and  470.  The  RFP  for  this  con- 
tract may  contain  a range  of  acceptable  R&M 
values,  so  that  the  potential  contractors  may 
propose  on  various  levels  within  that  range. 
The  contract  award  may  then  lie  made  at  the 
level  most  favorable  to  the  Air  Force  provided 
that  the  source  selection  criteria  define  the 
basis  for  the  award.  In  any  event,  the  final 
contract  must  contain  numerical  R&M  require- 
ments which  must  be  demonstrated  before  pro- 
duction begins. 

(4)  In  the  production  phase,  the  contract 
for  production  items  (including  reprocurement 
of  equipment  and  components  whose  R&M 
have  been  demonstrated  and  found  adequate) 
shall  include,  as  a minimum,  the  same  R&M 
values  for  the  production  equipment  and  shall 
require  periodic  verification  of  reliability  during 
production  runs.  Successful  R&M  demonstra- 
tion will  be  a condition  of  acceptance.  If  the 
R&M  parameters  of  the  existing  equipment 
have  been  found  inadequate,  the  procurement 
data  will  be  revised  to  reflect  the  required 
improvement.  Major  R&M  deficiencies  shall 
require  a special  program  for  improvement  and 
demonstration. 

(5)  In  the  production  phase,  the  procure- 
ment of  an  existing  item  which  did  not  have 
prior  R&M  requirements  imposed  but  which 
has  demonstrated  satisfactory  operational  R&M 
characteristics,  need  not  contain  provisions  for 
specific1  requirements;  however,  the  contract 
will  contain  provisions  to  assure  that  demon- 
strated R&M  characteristics  are  not  degraded. 
R&M  provisions  in  these  cases  must  be 
approved  by  the  R&M  focal  point  of  the  organi- 
zation involved.  The  supporting  rationale  will 
be  documented  in  the  contract  file. 

c.  Other  R&M  contractual  requirements 
include  the  following: 

(1)  R&M  demonstrations  shall  be  con- 
ducted as  an  integral  part  of  the  overall  test 
program  (AFR  80-14)  in  an  operational  or 
simulated  operational  environment  using  the 
test  procedures  prescribed  in  MIL-STDs  781 
and  471A.  When  the  procuring,  operating  and 
supporting  agencies  determine  jointly  that 
these  test  methods  are  inappropriate,  alternate 
tests  shall  be  specified  by  the  Air  Force.  In 
any  event,  the  procuring  activity  must  assure 
that  the  specified  tests  will  produce  adequate 
confidence  that  minimum  acceptable  levels  of 
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R&M  have  been  achieved.  The  demonstrations 
must  be  completed  before  the  end  of  the  full- 
scale  development  phase  and  the  results  made 
available  for  use  in  making  a production 
decision. 

(2)  When,  due  to  very  high  MTBF 
requirements,  meaningful  reliability  demon- 
strations cannot  be  performed  cost  effectively, 
design  qualification  may  be  achieved  by  an 
analysis  of  the  data  from  functional,  environ- 
mental, screening  and  low-confidence  reliability 
tests  of  the  item.  Similarly,  if  due  to  limited 
production  quantities,  meaningful  production 
acceptance  tests  are  not  practical,  production 
contracts  shall  include  reliability,  warranty  or 
correction  of  deficiency  provisions  covering  an 
appropriate  period  of  the  operational  phase. 
The  rationale  for  the  course  of  action  taken  will 
be  documented  in  the  contract  file. 

(3)  For  a less  complex  development  effort 
(that  is,  one  which  is  defined  as  an  individual 
equipment  or  minor  subsystem  development, 
involving  a limited  amount  of  new  and 
unproved  designs  and  characterized  by  no  con- 
ceptual or  validation  phases  preceding 
development),  a less  extensive  program  may  be 
used.  In  this  situation,  although  numerical 
R&M  requirements  and  demonstrations  are 
required,  selected  tasks  from  MIL-STDs  785 
and  470  may  be  used  without  invoking  the  com- 
plete standards  or  requiring  R&M  program 
plans. 

(4)  When  procurement  is  based  on  vendor 
technical  and  engineering  data  alone,  this  data 
must  provide  assurance  that  the  R&M  perform- 
ance parameters  are  adequate  for  the 
intended  application  and  are  controlled  for 
uniformity.  Generally,  when  an  off-the-shelf 
aircraft  or  item  of  aeronautical  equipment  is 
being  procured  that  has  been  certified  for  air- 
worthiness under  the  Federal  Air  Regulations, 
airworthiness  certification  test  records  and 
R&M  experience  during  civil  operations  will  be 
assessed  as  a basis  for  procurement.  The 
policies  outlined  in  paragraph  5b  will  apply  to 
complex  commercial  or  off-the-shelf  equipment, 
whose  functions  affect  safety  or  mission  accom- 
plishment as  follows: 

(a)  To  equipment  which  was  not  cer- 
tified to  operate  under  conditions  as  rigorous  as 
are  envisioned  for  its  USAF  use:  or 


(b)  To  uncertified  equipment  when  ven- 
dor technical  data  Ls  inadequate  as  determined 
by  Air  Force  engineering  personnel. 

(5)  The  contract  for  any  system  (or  part  of 
a system)  which  will  be  inaccessible  during  its 
operational  life,  or  which  employs  a “discard  at 
failure”  concept,  or  which  is  being  produced  by 
a manufacturer  other  than  the  designer,  shall 
not  require  a maintainability  program,  numeri- 
cal maintainability  requirements,  or  a main- 
tainability demonstration. 

(6)  Each  contract  schedule  shall  allow 
sufficient  time  for  satisfactory  demonstration  of 
the  minimum  acceptable  R&M  parameters  of 
hardware  involved. 

(7)  Each  contract  for  system  development 
and  acquisition  will  include  the  minimum  essen- 
tial data  requirements  upon  which  R&M  con- 
trol, visibility  and  management  decisions  may 
be  based.  The  standard  data  items  developed 
for  these  purposes  are  in  the  DOD  authorized 
Data  List  (DOD-ADI,). 

(8)  For  a major  weapon  system,  the  pro- 
curing activity,  with  the  aid  of  the  appropriate 
contract  management  agencies  (such  as  the  Air 
Force  Plant  Representative  Office),  will  main- 
tain surveillance  over  the  contractor’s  R&M 
programs.  For  other  than  a major  system,  the 
contract  administration  activity  will  maintain 
surveillance  over  the  contractor’s  R&M  pro- 
grams, but  will  provide  feedback  to  the  procur- 
ing activity  on  contractor  performance. 
However,  in  the  surveillance  of  any  contractor 
R&M  program,  the  procuring  activity  may 
obtain  support  from  any  Air  Force  activity  that 
has  the  pertinent  technical  capability,  or  from 
a technically  qualified  contractor  (if  no  conflict 
of  interest  is  involved). 

(9)  During  the  full-scale  development 
phase,  an  optimum  repair  level  analysis  will  be 
performed  in  accordance  with  the  joint  proce- 
dures of  AFLC  and  AFSC.  This  analysis  may 
be  performed  earlier  — as  a preliminary  study 
— but,  if  so,  it  will  be  updated  during  the  full- 
scale  development  phase. 

(10)  The  total  cost  for  the  delivery  of  a 
ready  and  operationally  effective  system  will 
include  the  cost  of  achieving  R&M.  Each 
change  in  the  original  scope  of  the  R&M  effort 
must  be  validated  by  a system  cost- 
effectiveness  analysis  that  encompasses  the 
total  life  cycle  of  the  system. 
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(11)  The  extent  to  which  this  regulation 
applies  to  quick  reaction  capability  (QRC)  pro- 
curements will  be  determined  under  APR 
57-5.  When  program  schedule  constraints  pre- 
clude use  of  a complete  R&M  program,  one  of 
reduced  degree  should  be  incorporated  such  as 
design  review,  bum-in,  R&M  analysis  and  pre- 
diction. If  R&M  requirements  are  included  in  a 
QRC  contract,  the  policies  outlined  in  this  reg- 
ulation will  be  used  for  guidance. 

fi.  Human  Performance  Reliability  Factors: 

a.  During  the  design,  development  and  test 
process,  R&M  program  monitoring  activities 
must  coordinate  closely  with  personnel  subsys- 
tem activities  (APR  80—46).  When  appropriate, 
the  policies  outlined  in  AFM  35-99  will  be 
applied. 

b.  The  human  performance  reliability 
analysis,  developed  by  implementing  APR 
80-46,  is  an  integral  part  of  system  R&M.  It 
includes  systematic  identification  and  recording 
of  critical  human  initiated  effects  in  relation- 
ship to  system  performance. 

7.  Test  and  Field  Data  System.  Air  Force 
test  and  field  data  systems  must  be  compatible 
and  be  capable  of  evaluating  and  verifying  sys- 
tem performance  throughout  its  life  cycle, 
including  in-depth  assessment  of  R&M  parame- 
ters. Each  data  system  must  provide  for  feed- 
back of  R&M  experience  to  influence  the 
design  of  new  systems  and  redesign  of  existing 
ones  where  necessary. 

8.  Engineering  Changes.  When  a system  does 
not  meet  test,  operation  or  technical  perform- 
ance requirements  because  of  R&M 
deficiencies,  engineering  changes  will  be  consid- 
ered. Approval  of  retrofit  changes  will  be 
accomplished  as  outlined  in  AFR  57-4. 

9.  Policy  Waivers.  Except  as  listed  below,  any 
request  for  a waiver  of  the  policy  outlined  here 
will  be  processed  through  command  channels  to 
HQ  USA  P/LG Y for  staffing  and  final  decision. 
Also,  HQ  USAP  approval  must  be  obtained 
before  scheduling  any  contract  program  that 
provides  for  a production  release  prior  to  the 
satisfactory  completion  of  R&M  testing  and 
analysis.  Any  of  the  waivers  or  deviations 
listed  below  require  the  approval  of  both  the 
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headquarters  of  the  major  command  accom- 
plishing the  procurement,  and  the  using  com- 
mand. This  authority  cannot  be  redelegated. 

a.  The  use  of  a nonstandard  MIL-STI)  test 
or  demonstration,  when  a standard  test  is 
available,  or  acceptance  by  similarity  of  prior 
tests  accomplished. 

b.  The  acceptance  of  test  or  demonstration 
results  that  are  less  than  required  by  contract. 

SEC  TION  C— RESPONSIBILITIES  FOR 
ESTABLISHING,  MONITORING  AND 
IMPLEMENTING  R&M  PROGRAMS. 

10.  Responsibilities  of  HQ  I’SAF.  This  head- 
quarters will: 

a.  Formulate,  establish,  and  maintain 
policies  on  R&M  programs  for  all  systems  and 
associated  material  in  design,  development, 
production,  and  operational  service. 

b.  In  each  PMD  include  the  R&M  require- 
ments or  objectives  (or  an  adequate  description 
of  each  mission-critical  parameter)  so  that 
meaningful  R&M  parameters  can  be  derived. 
Unless  expressly  waived  or  modified  in  the 
PMD,  the  policies  outlined  here  shall  apply. 

c.  Establish  an  Air  Staff  R&M  program 
OPR  and  an  R&M  focal  point  within  the 
appropriate  Air  Staff  elements,  to  monitor, 
review,  evaluate  and  guide  the  Air  Force  R&M 
program. 

d.  Direct  and  support  the  development  and 
establishment  of  an  Air  Force-wide  R&M 
educational  and  training  program  to  increase 
the  capability  of  professional,  technical,  and 
management  personnel  and  to  provide  for  an 
effective  exchange  of  technical  information. 

e.  Establish  and  maintain  a program  element 
(AFR  27-9)  for  general  R&M  technological 
improvement,  engineering  application,  and  con- 
trol. 

f.  Evaluate  major  command  requests  for  a 
waiver  of  R&M  policy  on  a specific  procure- 
ment or  development  effort,  and  advise  the 
requesting  command  of  the  decision  reached. 

11.  Responsibilities  of  the  Air  Force  Systems 
Command.  This  command  will: 

a.  Establish  and  monitor  a command  R&M 
program. 


I‘>? 


AFR  <S(> — 5 


2 July  1 !)":» 


b.  Establish  an  R&M  focal  point  within  the 
appropriate  staff  element  of  HQ  AFSC,  and  an 
office  of  primary  responsibility  (OPR)  at  other 
levels  of  command  organizations. 

c.  Designate  an  R&M  engineer  for  each  sys- 
tem program  or  project,  or  assign  a centrally 
located  R&M  engineer  to  each  program  to 
assist  the  director  or  manager  in  accomplishing 
the  following  R&M  responsibilities: 

(1)  Provide  R&M  inputs  into  program 
documentation  and  the  contractual  documents, 
according  to  the  policies  outlined  here. 

(2)  Be  responsible  to  the  system  program 
director  or  project  manager  for  the  attainment 
of  the  program's  R&M  requirements. 

d.  Support  the  operating  commands  in  estab- 
lishing R&M  objectives  or  in  identifying  critical 
mission  needs  so  that  R&M  parameters  can  be 
derived  for  inclusion  in  ROCs. 

e.  Conduct  research  and  development  pro- 
grams to  improve  techniques,  dealing  with 
specifying,  predicting,  demonstrating  and 
improving  R&M  for  developing  human  factors 
engineering  principles  and  design  criteria 
which  will  support  R&M  programs. 

f.  Incorporate  numerical  R&M  requirements, 
and  procedures  for  demonstrating  their 
achievement,  in  the  pertinent  specifications, 
exhibits,  product  descriptions,  work  state- 
ments, and  contractual  clauses  to  be  referred 
to,  or  included  in,  system  contracts.  Also 
assure  that  reprocurement  data  include  the 
numerical  R&M  requirements  for  use  in  replen- 
ishing a system’s  logistic  support. 

g.  Work  with  AFLC  and  the  operating  com- 
mands to  assure  that  the  contractor's  R&M 
program  proposal  is  evaluated  as  an  important 
part  of  systems  source  selection  action. 

h.  Assure  the  adequacy  of  each  contractual 
R&M  acceptance  test  and  demonstration. 
When  feasible,  use  an  integra  ed  test  plan  to 
minimize  the  cost  of  testing. 

i.  Implement  a data  system  to  measure  and 
evaluate  R&M  during  test  programs.  In  con- 
junction with  AFLC,  take  necessary  action  to 
implement  a "single  thread"  data  system. 

j.  Help  ATC'  and  AU  develop  formal  Air 
Force  R&M  education  and  training  programs 
and  courses.  Conduct  informal  R&M  training 


as  required  to  clarify  issues  and  resolve  misun- 
derstandings. 

k.  Coordinate  with  AFLC  and  the  using 
commands  to  assure  that  the  content  and  pre- 
sentation techniques  of  maintenance  and 
operating  instructions  are  compatible  with  the 
maintenance  concept  and  that  they  contribute 
proportionately  to  the  maintainability  of  the 
overall  system. 

l.  Include  development  reports  at  least  quar- 
terly in  each  program  acquisition  review  (PAR) 
on  the  status  of  system  R&M,  until  Air  Force 
engineering  responsibility  is  transferred  to 
AFLC. 


m.  Implement  a modification  program  to 
enhance  system  R&M  for  operational  hardware 
that  has  not  been  transferred  to  AFLC  for 
engineering  responsibility. 


n.  Transfer  to  AFLC  and  the  operating  com- 
mands the  R&M  data  (including  functional 
reliability  models)  obtained  and  verified  during 
development,  design,  test  and  evaluation. 

12.  Responsibilities  of  the  Air  Force  Logis- 
tics Command: 

a.  Establish  and  monitor  command  R&M 
programs  to  effectively  accomplish  command 
responsibilities. 


b.  Establish  R&M  focal  points  within  the 
appropriate  staff  element  of  HQ  AFLC  and 
OPRs  at  other  command  organizations  as 
necessary. 

c.  Help  Air  University  and  Air  Training 
Command  develop  Air  Force  R&M  education 
and  training  programs  and  courses.  Conduct 
command  education  and  training  programs  and 


courses. 

d.  Coordinate  R&M  requirements  in  AFSC 
developed  specifications  and  Statements  oil 
Work  for  equipment  destined  to  become  opera] 
tional  or  having  an  operational  application.  j 

I 

e.  Establish  or  update  R&M  requirement] 
for  items  procured  by  AFLC.  Where  engineer] 
ing  responsibility  lias  not  been  transferred 
coordinate  such  actions  with  AFSC. 


f.  Participate  in  source  selection  evaluation 
board  (SSEB)  action,  and  collaborate  with 
AFSC  and  the  operating  command,  to  deter- 
mine R&M  requirements  to  be  specified  in  the 
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RFP  and  the  work  statement  for  each  system 
that  is  destined  to  become  operational,  or  have 
an  operational  application. 

g.  Participate  in  the  design  review  of  new 
and  modified  systems  and  equipment.  Recom- 
mend alterations  or  changes  as  necessary  to 
improve  R&M  of  items. 

h.  With  AFSC.  and  the  using  commands, 
participate  in  tests  (APR  80-14)  to  evaluate 
achieved  R&M  and  support  provisions  for  new 
and  modified  systems. 

I , 

i.  With  AFSC,  identify,  develop  and  keep 
current,  a management  information  system  to 
reflect  the  degree  of  achieved  R&M,  using  oper- 
ational and  maintenance  data.  In  concert  with 
AFSC,  strive  for  achievement  of  a “single 
thread"  data  system. 

j.  Furnish  the  appropriate  R&M  manage- 
ment products  to  the  AFSC  and  the  operating 
commands. 

k.  Establish  an  R&M  performance  feedback 
system  to  support  the  effective  design  and  pro- 
duction of  new  systems. 

l.  Conduct  a maintenance  manager  review 
discussed  in  AFR  (16-12  and  adjust  inspection 
and  replacement  intervals  for  time  change 
items  when  justified  by  R&M  considerations. 

m.  With  acceptance  of  engineering  responsi- 
bility for  a system,  AFLC  will  accomplish  the 
following: 

(1)  Incorporate  or  revised,  as  appropriate, 
^ numerical  R&M  requirements  and  demonstra- 

tion provisions  in  applicable  specifications, 
exhibits,  product  descriptions,  work  statement, 
and  contractual  clauses  to  be  referred  to,  or 
included  in,  contracts  for  systems  and 
associated  material. 

- (2)  Perform  R&M  functions  related  to 

engineering  change  proposals,  modification 
programs,  inspection  procedures,  advanced 
procurement  data  programs,  qualification  and 
first  article  testing,  waivers  to  engineering  and 
technical  requirements,  and  system  interface 
problems. 

(8)  Determine  the  specific  reason  for 
unsatisfactory  hardware  R&M  and  initiate  cost 
effective  actions  to  increase  the  R&M  of  opera- 
tional systems  (AFRs  57-4,  66-30  and  400-46). 
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(4)  Accomplish  prototype  testing  and  suli 
by-side  operational  comparison  <>f  modified  v»*r 
sus  unmodified  hardware  (subsystem  and 
equipment)  as  necessary  to: 

(a)  Verify  the  forecasted  effectiveness  of 
the  modification  in  the  areas  of  reliability,  main 
tainability  and  logistic  sup|x>rt  cost. 

(b)  Verify  that  new  failure  modes 
associated  with  safety  art'  not  introduced. 


13.  Responsibilities  of  the  Operating  Com- 
mands. These  commands  will: 

a.  Establish  a headquarters  R&M  program 
manager,  or  panel,  to  develop  command  [xilicy 
for  implementing  Air  Force  policy. 

b.  Include  in  ROCs  and  CEIPs  submitted  to 
HQ  USAF,  planning  information  as  discussed 
in  paragraph  4 above. 

c.  Help  AFSC  prepare  and  review  system 
planning  documents  to  ensure  the  adequacy  of 
operational  and  logistic  sup|x>rt  requirements. 

e.  When  conducting  an  initial  operational 
test  and  evaluation  (OT&E)  of  new  systems 
and  equipment  (AFR  80-14),  verify  with  AFSC 
and  AFLC,  R&M  achievements  accomplished. 

f.  Refine  system  R&M  assessments  during 
follow-on  operational  test  and  evaluation  of 
new  systems  and  equipment  (AFR  80-14). 

g.  Help  AFLC: 

(1)  Conduct  formal  periodic  reviews  of 
R&M  programs  on  appropriate  operational  sys- 
tems. 

(2)  Collect  R&M  data  on  operationally 
deployed  systems  by  assigning  and  administer- 
ing necessary  resources  to  collect  and  transmit 
required  data. 

h.  Insure  that  operational  R&M  of  deployed 
systems  is  maintained  at  the  level  specified  in 
the  system  and  Cl  specifications;  report  hard- 
ware R&M  deficiencies  that  have  an  impact 
on  the  operational  capability  of  weapons  sys- 
tems or  require  excessive  maintenance 
resources  (manhours  and  costs);  participate  in 
a program  to  increase  the  R&M  of  operational 
systems  (AFR  400-46). 
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i.  Help  ATX'  and  AU  develop  Air  Force 
K&M  education  and  training  programs  and 
courses. 

j.  Conduct  command  K&M  education  and 
training  programs  and  courses. 

k.  Coordinate  K&M  requirements  in  A FSC 
and  AFLC  developed  specifications  and  state- 
ments of  work  for  equipment  destined  to  sup- 
port command  mission  requirements. 


a.  In  coordination  with  other  involved  com- 
mands, develop  and  conduct  Air  Force  K&M 
training  courses. 

1).  Evaluate  effectiveness  of  training  courses. 

15.  Responsibilities  of  the  Air  University. 

The  University  will: 

a.  In  coordination  with  other  involved  com- 
mands, develop  and  conduct  Air  Force  K&M 
education  programs. 


14.  Responsibilities  of  the  Air  Training  Com-  b.  Conduct  follow-up  evaluation  of  education 
mand.  This  command  will:  program  effectiveness. 


BY  ORDER  OF  THE  SECRETARY  OF  THE  AIR  FORCE 


OFFICIAL  JOHN  D.  RYAN,  JR,  General,  USAF 

Chief  of  Staff 

JACK  R.  BENSON,  Colonel,  USAF 
Director  of  Administration 

SUMMARY  OF  REVISED,  DELETED  OR  ADDED  MATERIAL 

This  revision  includes  the  phases  of  a system  life  cycle  and  program  documentation  defined  in  AFR 
800-2.  It  also  clarifies  the  distinction  between  hardware  reliability  and  operational  reliability  (para 
1);  provides  added  emphasis  to  the  need  for  cost  effectiveness  analysis  (para  3);  consolidates  in 
one  paragraph  all  of  the  information  about  contractual  documentation  and  requirements  (para  5): 
deletes  the  requirement  to  implement  a maintainability  program  when  not  appropriate  (para 
5c(5));  and  expands  and  clarifies  the  conditions  under  which  a policy  waiver  is  required  (para  9). 
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GENERAL 


AR  702-3 


1-1.  Purpose  and  applicability.  This  regulation — 

а.  Sets  forth  objectives,  concepts,  responsibili- 
ties,- anil  general  policies  for  Army  materiel  relia- 
bility, availability,  and  maintainability  (RAM) 
programs.  These  programs  apply  to  materiel 
systems,1  test  measurement  and  diagnostic  equip- 
ment (TMDE),  training  devices  and  facilities 
developed,  produced,  maintained,  procured,  or 
modified  for  Army  use  and  for  which  quantitative 
requirements  for  RAM  are  normally  applicable. 

б.  Includes  coverage  of  components  of  systems, 
TMDE,  training  devices,  and  facilities  to  the 
extent  that  they  are  integral  or  separate  functional 
components  of  military  systems  (e.g.,  generators, 
facilities  equipment,  missile  elevators,  etc.) 

c.  Establishes  the  manner  by  which  the  RAM 
characteristics  are  to  be  stated  in  requirements 
documents,  to  be  designed  into  systems,  to  be  used 
in  the  design  of  the  support  systems,  and  to  be 
assessed  throughout  the  life  cycle  of  materiel. 

d.  Includes  the  treatment  of  RAM  for  non- 
development items  (i.e.,  commercial  Nondevelop- 
mental  Items — CNDI,  Military  Adapted  Com- 
mercial Items — MACI,  and  equipment  developed 
by  other  military  services  or  allies). 

e.  Applies  to  the  Reserve  Components  when 
dealing  with  the  RAM  data  collection. 

1-2.  Objectives.  The  objectives  of  this  regulation 
are  to — 

a.  Insure  that  materiel  systems  provided  to 
the  Army  are  operationally  ready  for  use  when 
needed,  will  successfully  perform  their  assigned 
functions,  and  can  be  economically  operated  and 
maintained  within  the  scope  of  logistics  concepts 
and  policies. 

b.  Formulate,  define,  and  insure  promulgation  of 
RAM  requirements  for  systems  developed,  pro- 
cured, or  improved  to  meet  valid  user  needs 
(AR  70-15). 

c.  Assure  that  the  RAM  program  contributes  to 
reducing  the  overall  life  cycle  cost,  while  increasing 
the  overall  performance  etfectiveness  of  Army 
systems. 

1 The  term  system  is  used  throughout  this  regulation 
to  indicate  either  an  item  or  system  < group  of  items)  of 
materiel. 


d.  Assure  that  RAM  requirements  or  RAM 
characteristics  are  properly  demonstrated  and/or 
assessed  during  development  und  operational 
test  and  evaluation. 

e.  Assure  that  the  results  of  RAM  demonstra- 
tions and/or  assessments  are  portrayed  in  a 
suitable  form  for  the  decision  making  process 
(i.e.,  1 PR/ASA  RC/DSA  IiC) . 

1-3.  Concept,  a.  The  RAM  program  for  an  item  is 
pursued  within  the  framework  of  DA  Pam  11-25 
and  is  an  integral  part  of  the  general  system 
development  process  by  which  required  levels  of 
effectiveness  are  achieved  and  maintained  at 
minimum  cost  throughout  the  entire  life  of  the 
system. 

b.  The  effectiveness  of  a system  depends  on  its 
capability  and  availability  to  perform  a specified 
military  mission.  This  overall  effectiveness  de- 
pends on  the  materiel  design  capabilities,  including 
RAM,  and  the  concept  of  employment  and  sup- 
port. Consequently,  the  effectiveness  of  a system 
will  vary  according  to  its  reliability  and  main- 
tainability and  the  effectiveness  of  support  under 
multiple  s3'stem  uses  in  an  operational  environ- 
ment. 

c.  Appropriate  RAM  characteristics  or  higher 
order  effectiveness  parameters  (e.g.,  systems 
effectiveness,  annual  support  costs,  or  prob- 
ability of  single  shot  kill)  must  be  properly  and 
adequately  quantified  in  requirements  documents, 
included  in  contract  specifications,  and  measured 
during  tests,  in  order  to:  lnlluence  design  of  the 
system,  permit  an  accurate  determination  of  test 
requirements  and  test  results,  and  permit  logistic 
support  planning. 

d.  Achievement  of  RAM  requirements  will  be 
demonstrated  and  assessed  in  conjunction  with 
both  development  testing  (DT)  and  operational 
testing  (OT).  RAM  characteristics  will  be  assessed 
and  judged  as  to  their  adequacy  during  the  de- 
velopment process  by  Army  Systems  Acquisition 
Review  Council/Defense  Systems  Acquisition  Re- 
view Council,  and  In-Process- Review  (ASARC 
DSARC,  and  IPR)  (AR  15-14  and  AR  70-1) 
membership. 
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e.  Determining  RAM  requirements,  establishing 
contractual  RAM  values,  designing  RAM  tests, 
assessing  test  results,  and  decisions  concerning 
RAM  acceptability  are  separate  events  which 
require  separate  and  distinct  considerations. 

/.  Notwithstanding  the  provisions  of  this  regu- 
lation, all  users  of  this  regulation  (combat  de- 
velopers, materiel  developers,  testers,  and  evalu- 
ators) are  encouraged  to  exhibit  maximum  flexi- 
bility in  developing  and  executing  RAM  Programs 
or  any  subelement  to  Army  materiel.  RAM 
Programs  shall  be  tailored  to  the  individual  needs 
of  each  system.  RAM  programs  are  expected  to 
vary  in  scope  and  complexity  between  major  and 
nonmajor  systems.  Materiel  developers  should 
pursue  both  hardware  and  program  alternatives 
which  are  cost-effective  in  achieving  reliable  and 
maintainable  materiel. 

1-4.  Explanation  of  terms.  Terms  used  in  this 
regulation  are  explained  herein  and  in  Appendix 
A.  Also,  see  definitions  in  AR  310-25,  MIL-STD- 
721B,  and  MIL-STD-781B,  and  abbreviations  in 
AR  310-50. 

1-5.  Responsibilities.  The  AR  10  series  describes 
responsibilities  of  the  Army  Staff  and  of  the  major 
Army  commands.  The  AR  70  series,  AR  702  series, 
and  AR  750  series  describe  specific  responsibilities 
in  Army  research  and  development,  product 
assurance,  and  maintenance.  Additional  specific 
responsibilities  for  carrying  out  the  RAM  program 
are  described  below. 

a.  Deputy  Chief  of  Staff  for  Research,  Develop- 
ment, and  Acquisition  (DCSRDA). 

(1)  Has  primary  Army  general  staff  responsi- 
bility for  the  overall  RAM  program  pertaining  to 
materiel. 

(2)  Issues  Army  policies  on  reliability, 
availability,  and  maintainabilty. 

(3)  Supervises  the  major  RAM  program 
elements  to — 

(а)  Insure  that  valid  RAM  requirements 
are  addressed  in  the  requirements  documentation. 

(б)  Insure  that  Development  Plans  (AR 
70-27)  provide  for  achievement,  assessment,  and 
testing  (AR  70-10)  of  RAM. 

(c)  Promote  development,  improvement, 
and  application  of  RAM  technology  and  design 
practices. 

(d)  Insure  that  RAM  is  evaluated  in 
product  improvement  programs  (AR  70-15). 
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(4)  Monitors  consideration  of  RAM  during 
the  DA  decision  making  process. 

b.  Deputy  Chief  of  Staff  for  Logistics 

(DCSLOG). 

(1)  Insures  continued  compatibility  between 
the  Integrated  Logistic  Support  (ILS)  program 
(AR  700-127)  and  the  RAM  program. 

(2)  Provides  policy  guidance  concerning  the 
provision  of  logistic  support  input  to  combat 
development  and  materiel  development  agencies 
for  use  in  developing  and  validating  RAM 
characteristics  on  new  materiel. 

(3)  In  coordination  with  ODCSOPS, 
ODCSRDA,  and  OTEA,  has  approval  authority 
for  deviation  from  complete  maintenance  test 
support  packages  for  DT/OT  II  and  III  for  major 
and  category  1 nonmajor  systems. 

(4)  Insures  continued  data  collection  and 
assessment  of  RAM  performance  for  deployed 
systems. 

(5)  Assists  in  DA  Staff  evaluation  of  changes 
to  operational  materiel  RAM  characteristics 
resulting  from  product  improvement  programs. 

c.  Deputy  Chief  of  Staff  for  Personnel  (DCS- 
PER),  Deputy  Chief  of  Staff  for  Plans  and 
Operations  (DCSOPS),  Assistant  Chief  of  Staff 
for  Intelligence  (ACSI) , Chief  of  Engineers  (COE) 
and  the  Surgeon  General.  Implement  the  policies 
set  forth  in  their  respective  areas  of  interest. 

d.  Commanding  General,  US  Army  Operational 

Test  and  Evaluation  Agency.  For  assigned  items 
of  materiel 

(1)  Provides  RAM  OT  portion  of  CTP, 
review's  and  concurs  in  CTP’s. 

(2)  Conducts  operational  tests  and  evalua- 
tions to  assess  RAM. 

(3)  Reviews  and  comments  or  coordinates 
on  documents  pertinent  to  RAM  aspects  of  OT. 

e.  All  materiel  developing  and  procuring 
agencies. 

(1)  Determine  jointly  with  the  combat  de- 
veloper realistic  RAM  characteristics  consistent 
with  system  operational  and  support  require- 
ments, current  state-of-the-art,  Army  doctrine, 
organization  and  force  structure,  and  the  COEA. 

(2)  Validate  the  technical  feasibility'  of  RAM 
requirements  proposed  for  items  under  the 
agency’s  development. 

(3)  Establish  and  maintain  integrated  con- 
trols to  insure  achievement  of  RAM  require- 
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merits  and  execution  of  the  policies  of  this 
regulation. 

(4)  Assess  critical  elements  of  RAM  through- 
out the  life  cycle  to  detect  causes  or  trends  which 
may  degrade  system  performance,  operational 
readiness,  or  increase  life  cycle  costs,  and  propose 
or  take  corrective  action. 

(5)  Coordinate  with  the  appropriate  major 
Army  commands  for  on-site  monitoring  of  tactical 
materiel. 

(6)  Insure  that  a RAM  data  base  is  main- 
tained (consistent  with  standard  data  systems) 
for  materiel  under  their  responsibility. 

(7)  Have  approval  authority  for  deviation 
from  complete  maintenance  test  support  packages 
for  DT  and  OT  conducted  on  category  2,  3,  and 
4 systems;  following  coordination  with  members 
of  the  inprocess  review  (IPR).  (See  AR  700-127.) 

/.  Commanding  General,  US  Army  Materiel 
Development  and  Readiness  Command.  In  addi- 
tion to  the  responsibilities  in  paragraph  1— 5c : 

(1)  Maintains  a central  activity  for  developing 
and  improving  RAM  program  methodologies  by — - 

(а)  Disseminating  information  concerning 
new  RAM  techniques  applicable  to  Army  pro- 
grams. 

(б)  Identifying  related  technology  gaps  and 
recommending  supporting  research. 

(c)  Maintaining  effective  contact  with  ele- 
ments of  other  government  agencies,  private  in- 
dustry, and  universities  on  current  technical 
progress  in  the  state-of-the-art. 

(2)  Determines  requirements,  in  coordination 
with  other  activities,  for  developing  and  carrying 
out  Army  reliability  and  maintainability  education 
and  training  programs. 

(3)  Determines  requirements,  in  coordination 
with  other  activities,  for  publishing  and  maintain- 
ing Army  RAM  handbooks  and  pamphlets. 

(4)  Executes  staff  proponency  for  AR  702-3. 

g.  All  combat  developing  activites. 

(1)  Establish  and  maintain  controls  to  insure 
effective  coordination  of  RAM  program  functions 
and  execution  of  the  policies  of  this  regulation. 

(2)  Determine  jointly  with  the  materiel  de- 
veloper realistic  RAM  characteristics  consistent 
with  system  operational  and  support  requirements, 
current  state-of-the-art,  Army  doctrine,  organi- 
zation and  force  structure,  and  the  COEA. 

(3)  Monitor  materiel  development  to  ascer- 
tain the  degree  to  which  the  system  has  met  the 
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RAM  requirements  as  demonstrated  during  appro- 
priate testing. 

(4)  Monitor  RAM  performance  throughout 
the  life  cycle  of  materiel. 

(5)  Are  responsible  for  the  logistic  system 
certification  as  it  impacts  on  RAM. 

(6)  Are  responsible  for  the  training  certifica- 
tion as  it  impacts  on  RAM. 

h.  Commanding  General,  US  Array  Training 
and  Doctrine  Command.  In  addition  to  responsi- 
bilities in  paragraph  1-5 g: 

(1)  Maintains  a central  activity  for  the  proper 
statement  and  content  of  RAM  characteristics  in 
materiel  requirements  documentation. 

(2)  Keeps  abreast  of  current  progress  on  state- 
of-the-art  in  RAM  methodology  as  applied  to 
combat  developments. 

(3)  Conducts  OT  on  assigned  items  of  ma- 
teriel to  assess  RAM  and  provides  RAM  OT 
portion  of  CTP. 

i.  Logistic  (Readiness)  agencies  having  national 
maintenance  points  (AR  750-1).  Establish  and 
maintain  controls  to  insure  that — 

(1)  Maintenance  and  logistic  support  concept 
and  planning  information  is  coordinated  with  the 
development  activity  responsible  for  equipment 
design. 

(2)  Planning  for  maintenance  and  other  logis- 
tic support  is  compatible  with  specified  RAM  de- 
sign requirements. 

(3)  Reliability  and  maintainability  charac- 
teristics are  maintained  or  improved  during  prod- 
uct improvement  of  materiel  (AR  70-15). 

US  Army  Logistics  Evaluation  Agency 
(USALEA). 

(1)  Provides  surveillance  of  logistics  support 
implications  of  proposed  new  items  of  materiel. 

(2)  Evaluates  all  developmental  and  non- 
devolopmental  items  for  logistics  support  ability 
aspects  of  RAM  in  performance  of  its  mission  as 
the  Logistician  (AR  70-1). 

1-6.  Coordination,  a.  Logistics.  Operational  RAM 
can  be  significantly  altered  by  changes  in  opera- 
tional, environmental,  or  logistical  support  con- 
cepts. Organizations  concerned  with  the  RAM 
of  a system  and  its  components  must  maintain 
close  coordination  throughout  the  system’s  life 
cycle  with  organizations  responsible  for  the 
operation  and  logistic  support  of  materiel.  This 
coordination  should  assure  that  RAM  charnc- 
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teristics  nre  mutually  compatible  with  logistic 
concepts.  Maintenance  concepts,  repair  parts 
provisioning,  and  allocation  of  maintenance  re- 
sources must  support  the  system  readiness  require- 
ment. The  RAM  program  must  interface  with 
logistic  support  planning  and  execution  to  insure 
that  each  complement  one  another  and  enhance 
the  achievement  of  an  affordable  and  supportable 
system  that  performs  its  intended  function. 

6.  Personnel  and  training  Junctions.  Operational 
RAM  characteristics  are  interrelated  with  human 
performance  requirements  and  the  development 
and  use  of  trained  personnel.  Throughout  the 
life  cycle,  coordinated  planning  among  using, 
developing,  logistics,  personnel,  and  training 
agencies  is  required  to  assure  compatibility  be- 
tween quantitative  and  qualitative  personnel 
resources  and  materiel  readiness  requirements. 

c.  Defense  Standardization  Program  (DODD 
4110.3).  Developing  and  procuring  agencies 
should  participate  in  the  Defense  Standardiza- 
tion Program  through  the  preparation  and  use  of 
applicable  military  specifications,  standards, 
hand-books,  and  standardization  studies  in  order 
to  enhance  interchangeability,  reliability,  and 
maintainability  of  military  equipment  and  sup- 
plies. (See  AR  700-47.) 

d.  Delated  regulations.  Regulations  which  give 
additional  information  on  the  duties  and  re- 
sponsibilities delineated  in  this  regulation  arc 
listed  in  appendix  B. 

e.  Test  measurement  and  diagnostic  equipment 
ITMDE).  Use  of  TMDE  and  Built-in-Test 
Equipment  (BITE)  for  hardware  systems  diag- 
nosis or  prognosis  will  be  given  consideration 
during  design  and  development.  TMDE  require- 
ments, except  COMSEC  materiel,  will  be 
coordinated  with  Commander,  US  Army  Mainte- 
nance Management  Center,  Lexington-Bluegrass 
Army  Depot,  ATTN:  DRXMD-T,  (TMDE 
Product  Management  Office),  Lexington,  KY 
40507,  to  insure  maximum  use  is  made  of  items 
developed  and  available  to  the  Army,  (AR 
750-43).  TMDE  requirements  for  COMSEC 
materiel  will  be  coordinated  with  Commander, 
US  Army  Communications  Security  Logistics 
Agency,  Fort  Huachuca,  AZ  85013. 

j.  Support  documentation  and  analysis.  Pro- 
cedures will  be  established  to  insure  that  RAM 
data  are  compatible  with  logistic  support  analysis 
requirements  (AR  700-127). 
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1-7.  Reliability  and  Maintainability  Summary 
Report,  Requirement  Control  Symbol:  CSCRD-73. 

a.  Applicability.  Reports  will  be  prepared  by  the 
following: 

(1)  Commanding  General,  US  Army  Materiel 
Development  and  Readiness  Command. 

(2)  Commanding  General,  US  Army  Training 
and  Doctrine  Command. 

(3)  Commanding  General,  US  Army  Security 
Agency. 

(4)  Commanding  General,  IIS  Army  Com- 
munications Command. 

(5)  Commanding  General,  US  Army  Opera- 
tional Test  and  Evaluation  Agency. 

b.  Frequency.  Report  will  be  submitted  annually, 
as  of  the  end  of  each  fiscal  year. 

c.  Submission.  Five  copies  of  each  report  will 
be  submitted  to  IIQDA  (DAMA-PPM-T)  Wash- 
ington, DC  20310,  NLT  30  days  after  the  end  of 
each  reporting  period. 

(/.  General.  This  report  will  be  prepared  in  a 
narrative  letter  form.  Inclosures  may  be  used. 
This  report  not  only  provides  information  as  to 
program  status  but  will  also  identify  program, 
hardware,  methodology,  and  software  improve- 
ments made  by  one  Army  command  or  agency, 
which  may  be  applicable  to  others.  Each  reporting 
activity  will  submit  information  covering  only 
those  subjects  pertinent  to  its  efforts. 

(1)  Hardware  improvement  section.  Selected 
examples  of  hardware  improvements  during  the 
equipment  life  cycle  stages  of  development,  pro- 
duction, and  operational  use  will  be  given.  The 
format  should  identify  the  item  of  hardware;  the 
before  and  after  RAM  values  for  completed  im- 
provements; the  area  of  improvement;  and  related 
benefits  of  the  improvement,  such  as  increased 
availability,  estimated  5-year  cost  avoidance, 
decreased  logistical  burden,  and  related  per- 
formance payoffs. 

(2)  Program  improvement  section.  Summaries 
will  be  given  on  recently  initiated  RAM  program 
improvements,  including  the  following: 

(а)  RAM  training  courses. 

(б)  Improvements  in  timeliness  and  effec- 
tiveness of  failure  reporting,  and  analysis  of  critical 
and  corrective  actions. 

(c)  Strengthened  policies,  procedures,  and 
management  controls  having  an  impact  on  the 
effectiveness  of  the  RAM  program. 
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((/)  Now  techniques  and  procedures  for  will  normally  be  limited  to  one  page.  Inclosures 
prediction,  design,  and  assessment  of  RAM  may  be  used. 

characteristics.  (4)  Methodological  summary  section.  Sum- 

(e)  Significant  activities  which  have  ad-  maries  will  be  provided  for  unique  and  new 
vanced  RAM  achievement.  applications  of  methodological  approaches  for 

(3)  Report  abstracts  section.  Abstracts  will  assessment  of  RAM.  Limitations  or  weaknesses 

be  provided  for  reports  on  research,  development,  in  existing  methodology  will  be  identified, 
production,  or  maintenance  projects  specifically  (5)  Recommendations  section.  Recommenda- 

related  to  materiel  RAM;  physics  of  failure;  tions  should  be  made  a-  '<>  changes  needed  in  the 

safety;  test  techniques;  and  diagnostic  test  equip-  Army  RAM  progr to  impiove  the  capability 

rnent  (in-house  or  contracted).  The  abstracts  of  achieving  or  retaining  desirable  RAM  le\ cl-. 
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POLICIES 
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Section  I.  GENERAL  POLICIES 


2-1.  Management.  Planning,  programing,  and 
resource  allocation  will  be  provided  throughout 
the  life  cycle  of  each  system  to  achieve  the  11AM 
hardware  and  program  objectives  and  require- 
ments. The  impact  of  RAM  on  mission  accomplish- 
ment and  logistic  support  will  be  considered. 
RAM  characteristics  will  be  identified  among  the 
principal  performance  characteristics  of  the  sys- 
tem and  will  be  reviewed  by  management  and 
assessed  throughout  all  life  cycle  phases. 

2-2.  Life  Cycle  RAM  Program,  a.  Conceptual  phase. 
During  the  feasibility  studies  and  component 
development,  the  RAM  potential  of  the  item  and 
its  components  will  be  examined.  RAM  evalua- 
tions will  be  conducted  to  assist  in  selecting  proper 
technical  approaches  and  to  identify  high  technical 
risk.  This  information  will  also  be  used  in  the  trade- 
off and  cost  effectiveness  studies  and  to  assist  in 
final  concept  selection.  RAM  data  collected  from 
previous  items,  as  well  as  data  generated  during 
research  and  feasibility  studies,  will  be  used  and 
carried  forth  into  the  subsequent  phases. 

h.  Validation  phase.  RAM  efforts  during  the 
validation  phase  will  be  structured  towards  de- 
veloping RAM  requirements  which  are  credible, 
realistic,  attainable,  meet  an  operational  need  and 
contribute  to  minimizing  life  cycle  costs.  Con- 
sequently, the  relative  importance  of  various 
RAM  parameters  for  the  item  will  be  established 
during  this  phase  along  with  the  impact  of  RAM 
on  operational  performance  and  on  logistics 
suppor lability.  Quantitative  RAM  goals  or  ob- 
jectives (i.e.,  not  firm  requirements)  may  be 
established  and  used  during  the  validation  phase. 
Based  on  this,  the  degree  to  which  RAM  may7  be 
traded  off  will  also  be  ascertained.  Typical 
RAM  activities  such  as  predictions,  analyses, 
data  collection  and  assessment  and  failure  analysis/ 
corrective  action  will  be  executed.  Firm  RAM 
requirements  will  be  established  during  this 
phase  for  inclusion  in  the  ROC,  LR,  or  TDR. 

c.  Full  scale  development  phase.  RAM  program 
activities  and  management  controls  will  be  exe- 
cuted and  exercised  in  order  to  insure  that  materiel 
is  developed  which  fully  conforms  with  the  stated 
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requirements.  Quantitative  RAM  requirements  in 
a Required  Operational  Capability  (ROC),  'fram- 
ing Device  Requirement  (TDR),  or  Letter  Re- 
quirement (LR)  will  be  included,  and  expanded  as 
necessary,  in  the  Development  Plan.  RAM 
requirements,  programs,  testing  management  con- 
trols, and  data  systems  will  be  included  in  solici- 
tations and  contracts.  RAM  interfaces  with  other 
engineering  functions  will  be  established  to  insure 
compatibility  and  smooth  production  transitions. 

d.  Production  and  deployment.  RAM  efforts 
during  production  will  be  oriented  towards  in- 
suring the  minimum  degradation  of  RAM  during 
the  transition  to  and  continuation  of  production 
and  correction  of  deficiencies  found  during  testing. 
As  a part  of  the  System  Assessment  and  RAM 
Improvement  of  Selected  Equipment  (RISE) 
activities,  RAM  data  will  be  collected  and  ana- 
lyzed throughout  production  and  deployment  to 
assure  that  RAM  levels  are  maintained,  correc- 
tive actions  are  initiated,  and  RAM  product 
improvement  candidates  are  identified. 

2-3.  Reliability  growth,  a.  Reliability  of  materiel 
should  increase  during  the  development  process. 
Such  “reliability  growth”  is  a function  of  the 
maturity  of  design  and  the  application  of  en- 
gineering and  test  resources.  The  Army  objective 
of  delivering  systems  that  have  good  reliability 
characteristics  cannot  be  accomplished  simply 
through  testing  systems  at  the  end  of  the  develop- 
ment cycle.  Decision  bodies  must  be  provided 
visibility  on  howT  reliability  is  progressing  through- 
out the  program  so  that  cost  effective  action  and 
corrections  can  be  made  prior  to  the  production 
decision  on  the  item/system. 

b.  Materiel  developers  will  use  reliability  growth 
management  to  insure  visibility  of  system  re- 
liability progress  during  the  development  cycle. 
This  reliability  growth  management  will  consider 
not  only  the  growth  to  the  specified  value  but  will 
also  aid  in  making  a determination  of  the  time  and 
resources  necessary  to  reach  this  requirement. 
Further,  intermediate  reliability  criteria  will  be 
established  from  the  growth  curve  for  the  system 
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to  assist  the  materiel  developer  in  assessing 
progress  toward  the  specified  value. 

c.  Materiel  developers  will  develop  and  use 
reliability  growth  curves  on  all  major  and  desig- 
nated nonmajor  systems. 

d.  Reliability  growth  as  used  in  the  Army 
Materiel  Acquisition  Decision  Process  is  a manage- 
ment tool  rather  than  a technical  tool.  Reliability 
growth  management  should  provide  the  following 
to  the  decision  process: 

(1)  Aid  in  allocating  resources  to  achieve 
reliability  requirements  on  schedule  anti  within 
cost  constraints. 

(2)  Focus  attention  on  failure  to  meet  inter- 
mediate criteria  in  order  that  corrective  actions 
can  be  taken  to  meet  the  final  reliability  require- 
ment. 

(3)  Place  into  perspective  the  relationship 
of  achievement  of  required  RAM  to  the  rest  of 
the  development  and  testing  program. 

(4)  Indicate  early  in  the  development  cycle, 
preferably  by  DT/OT  I,  but  no  later  than  DT/OT 
II,  the  achievability  of  the  specified  value  for 
reliability. 

2-4.  Data  collection,  a.  RAM  data  are  required 
during  all  phases  of  the  materiel  life  cycle.  During 
development  and  operational  testing,  RAM  data 
are  required  to  evaluate  the  materiel  system  and 
plan  for  its  support.  During  production  and 
deployment,  RAM  data  are  required  to  evaluate 
the  performance  of  materiel  systems  and  major 
subsystems,  provide  a foundation  for  future 
development  programs  and  provide  for  logistic 
support  information.  Data  will  be  collected  in 
consonance  with  AR  700-127  and  MIL-STD- 
1388-1  and  -2. 

b.  An  effective  and  economical  RAM  data  sys- 
tem will  be  established  by  each  activity  having 
a materiel  development  or  procurement  mission. 
All  data  systems  will  provide  for  organized  and 
timely  collection,  analysis,  use  (immediate  and 
long  range),  systematic  storage,  and  disposition 
of  the  data.  The  data  system  will  include  all  data 
needed  to  permit  assessment  of  the  RAM  char- 
acteristics of  an  item  and  its  components.  Data 
systems  must  be  based  on  plans  (included  in 
development  plans)  which  define  exactly  what 
data  elements  are  to  be  collected  and  the  controls 
to  be  exercised  to  insure  that  all  data  are  appro- 
priately disseminated,  analyzed,  and  used  expedi- 
tinn-lv.  The  data  system  will  be  established  and 


maintained  to  meet  the  needs  of  its  users  ''e.g., 
combat  developers,  materiel  developers,  testers, 
and  evaluators). 

c.  All  RAM  data  and  incident  reports  from  DT 
and  OT  will  be  provided  and  exchanged  among 
the  combat  developer,  materiel  developer,  opera- 
tional tester,  and  logistician  on  a timely  ami 
responsive  basis.  Complete  and  detailed  data 
collection  plans,  procedures,  forms,  and  incident 
reporting  procedures  will  be  coordinated  among 
the  four  parties  at  a pretest  conference  to  insure 
that  all  data  needs  are  fulfilled. 

d.  A field  RAM  engineering  data  collection 
system  (e.g.,  controlled  data  collection)  will  be 
planned,  budgeted,  and  executed  to  provide  essen- 
tial information  for  a field  feedback  and  corrective 
action  loop  and  for  baseline  data  to  evaluate 
potential  requirements  for  future  systems.  A<  a 
minimum,  RAM  engineering  data  will  be  collected 
and  maintained  on  major  and  selected  nonmajor 
materiel  systems.  For  all  others,  the  extent  of  data 
collection  after  deployment  will  be  determined  by 
the  materiel  developer  and  as  requested  by 
HQDA  or  the  combat  developer  to  support 
specific  requirements. 

e.  Data  information  exchanges,  like  the  Govern- 
ment-Industry Data  Exchange  Program  (GIDEP), 
should  be  used  whenever  feasible.  Additional 
information  on  GIDEP  can  be  obtained  from  the 
Officer  in  Charge,  GIDEP  Operations  Center, 
Corona,  CA  91720. 

2-5.  Audit  trail,  a.  Materiel  developers/procurers 
will  establish  an  audit  trail  in  consonance  with 
AR  700-127  to  insure  traceability  of  essential 
RAM  and  support  parameters  at  any  time  during 
the  materiel  system’s  life  cycle. 

b.  The  materiel  developer  will  establish  specific 
controls  and  followup  procedures  to  insure  that 
deficiencies  discovered  during  all  testing  (DT  and 
OT)  are  corrected  and  that  an  audit  trail  is 
maintained.  Deficiencies  which  occur  during  early 
testing,  for  example  DT/OT  I,  should  not  recur 
during  DT/OT  II  or  DT/OT  III  testing.  The 
controls  and  audit  trail  established  by  the  materiel 
developer  will  insure  that  the  status  of  the  cor- 
rection of  known  deficiencies  is  periodically 
reviewed  from  the  end  of  testing  through  the  next 
major  test  and  decision  review.  When  DT/OT  111 
testing  is  completed,  the  materiel  developer  will 
take  action  either  on  his  own  accord  or  as  directed 
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by  the  decision  authority  (1PR/ASARC/DSA  RC) 
to  correct,  prior  to  deployment,  those  deficiencies 
which  affect  mission  performance.  The  corrections 
will  he  incorporated  into  production  items  / 
systems,  tested,  and  verified. 

c.  Verification  that  the  item  or  system  ns  modi- 
fied by  the  materiel  developer  has  had  HAM 
deficiencies  corrected  will  be  provided  by  the 
operational  tester  before  initial  issue  to  Army 
units. 

d.  The  combat  developer  will  determine  whether 
the  RAM  levels  of  performance  of  the  items  or 
system  under  test,  after  the  corrections  are 
applied,  meet  all  applicable  requirements. 

e.  The  independent  evaluator  for  OT  will 
determine  through  independent  evaluations  (re- 
views) of  tests,  whether  these  corrections  have 
insured  that  the  tested  item  or  system  has  achieved 
the  necessary  RAM  levels. 

j.  The  materiel  developer  will  document  these 


assessments  by  establishing  and  maintaining  a 
RAM  record  on  all  major,  nonmajor,  and  other 
designated  systems  throughout  the  materiel  lift* 
cycle.  This  record  will  provide  for  both  the  current 
status  and  the  historical  audit  trail  of  RAM. 
'This  record  should  include  such  information  as; 
RAM  requirements  and  their  rationale,  the  Failure 
Definition/Scoring  Criteria  (FD/SC),  the  Opera- 
tional Mode  Summary/Mission  Profile  (OMS 
Ml’),  tire  RAM  program  plan  of  both  Government 
and  contractor  RAM  activities,  the  contractual 
RAM  provisions,  all  test  results  (i.e.,  type  of 
test,  test  condition,  length,  failures,  data  analysis, 
problems,  and  corrective  actions),  current  and 
historical  growth  curves,  development  RAM 
status,  production  RAM  levels,  and  demonstrated 
field  RAM  characteristics.  The  record  should  he 
a self-standing  document  that  is  added  to  and 
updated,  and  thus  becomes  a living  RAM  audit 
trail. 


Sectioh  II.  DOCUMENTATION 


2-6.  Requirements  documents,  a.  RAM  charac- 
teristics. RAM  characteristics  or  other  suitable 
measures  of  effectiveness  will  be  stated  in  require- 
ments documentation  when  deenu  d appropriate 
by  the  combat  and  materiel  developers.  These 
requirements  will  be  stated  in  terms  appropriate 
for  the  system,  considering  its  intended  purpose, 
complexity,  method  of  acquisition,  and  cost  effec- 
tiveness. They  will  be  clear,  quantitative,  and 
capable  of  being  measured,  tested  or  Otherwise 
verified.  The  attributes  described  and  the  depth 
of  coverage  in  each  document  will  depend  on  the 
peculiarities  of  the  system  and  the  progress  of 
development.  Requirements  documents  will  slate 
only  those  principal  performance  and  RAM 
characteristics  which  are  essential  to  describe 
the  operational  features  of  the  system.  Care  must 
he  taken  to  insure  that  the  requirements  are  truly 
essential  and  do  not  preclude  viable  trade-offs. 
These  requirements  will  represent  the  combat  and 
materiel  developers  best  estimate  of  what  is 
needed  and  attainable. 

b.  Letters  oj  agreement  (Uhl).  For  those  pro- 
grams for  which  it  has  been  determined  that 
quantitative  RAM  requirements  will  he  stated 
in  a subsequent  Required  Operational  Capability 
(ROC),  Letter  Requirement  (LR),  or  TDK,  the 
LOA  must  outline  the  RAM  ta^ks,  assignments, 
and  testing  issues  that  must  be  accomplished  or 
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addressed  during  advanced  development  to  ade- 
quately quantify  RAM  for  full-scale  development. 
By  the  time  a ROC  or  LR  is  approved  for  the 
commencement  of  full-scale  engineering  develop- 
ment, sufficient  knowledge  must  be  known  from 
the  advanced  development  efforts  governed  by 
the  LOA  to  develop  and  justify  quantitative  RAM 
requirements  in  the  ROC,  LR,  or  TDR  when  it 
is  prepared.  Tims,  the  following  apply  to  LOA's : 

(1)  Normally,  the  LOA’s  will  not  contain 
quantitative  RAM  requirements.  However,  when 
deemed  necessary  or  advantageous,  quantitative 
RAM  goals  or  objectives  may  be  stated. 

(2)  The  LOA  will  contain  a clear  statement  as 
to  whether  or  not  future  quantitative  RAM 
requirements  will  be  included  in  a subsequent 
ROC  or  LR. 

(3)  When  the  LOA  states  that  quantitative 
RAM  requirements  will  be  established,  then  the 
LOA: 

(a)  Will  contain  a statement  which  rommit- 
tlie  combat  and  materiel  developers  to  initiate 
and  complete  work  on  tin1  item’s  operational 
mode  summary,  mission  profile  (OMS/MP),  relia- 
bility failure  definition/scoring  criteria  (FI), SC  , 
definitions  of  availability  and  maintainability 
parameters  and  a baseline  analysis  of  all  relevant 
RAM  data.  This  information,  in  conjunction  with 
other  cost,  technical  and  operational  inputs  will 
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be  subsequently  used  to  develop  the  RAM 
requirements. 

(b)  Will  state  whether  reliability,  availabil- 
ity, and,  or  maintainability  will  be  critical  issues 
for  testing.  A critical  RAM  testing  issue  estab- 
lishes that  issue  as  a testing  focal  point  and 
identifies  the  need  to  collect  and  analyze  data  on 
this  issue  to  provide  needed  information.  For 
example,  RAM  will  be  a critical  issue  during 
early  development  testing  if  inadequate  data 
exist  to  establish  a RAM  baseline. 

(o)  May  state  quantitative  RAM  goals  or 
objectives.  If  stated,  these  goals  or  objectives  will 
guide  the  advanced  development  effort  but  will 
not  be  treated  as  firm  requirements. 

(4)  When  quantitative  RAM  requirements 
will  not  be  stated  in  a subsequent  ROC  or  LR, 
then  the  LOA  should  briefly  address  the  rationale 
for  this  and  state  what,  if  any,  RAM  activities 
are  to  be  executed. 

c.  HOC’s,  IJi’s,  or  training  device  requirements 
(TDH’s)  for  developmental  systems. 

(1)  Jointly  agreed  to  quantitative  RAM 
requirements  must  be  stated  in  the  ROC,  LR, 
or  TDR,  unless  jointly  determined  bv  the  combat 
and  materiel  developers  that  RAM  requirements 
are  not  appropriate  for  the  item.  RAM  require- 
ments will  be  stated  in  accordance  with  one  of  the 
following  categories. 

(а)  RAM  not  required  category.  This  cate- 
gory will  be  used  when  it  can  be  determined  that 
RAM  characteristics  are  not  essential  or  applicable 
to  an  item  of  equipment  or  are  included  in  a higher 
order  effectiveness  parameter  (e.g.,  systems  effec- 
tiveness). In  those  cases  where  it  is  determined 
that  a statement  of  RAM  requirements  is  not 
applicable,  a brief  statement  to  this  effect,  with 
rationale,  will  be  included  in  (lie  rationale  annex. 

(б)  HAM  quantitative  requirements  category. 
This  category  is  used  when  specified  numbers  are 
provided.  Normally,  one  of  the  following  sub- 
categories  i<  sufficient  to  express  the  user’s  need: 

1.  Availability  and  reliability  require- 

ments. 

2.  Reliability  and  maintainability  re- 

quirements. 

(2;  When  either  of  these  RAM  categories  is 
used , the  following  statement  will  precede  the 
requirements  statement.  “The  quantitative  RAM 


requirements  contained  in  this  document  repre- 
sent the  best  estimate  of  the  operational  and  tech- 
nical requirements  for  this  system  based  upon 
currently  available  knowledge.  However,  when 
information  is  gained  from  subsequent  studies, 
trade-off  analyses  and  cost  effectiveness  evalua- 
tions that  indicate  a change  in  the  threat,  need, 
or  operational/tcchnieul  capabilities,  the  combat 
and  materiel  developers  will  jointly  initiate  a 
change  to  the  appropriate  RAM  requirement.” 

(3)  When  RAM  requirements  are  appropriate 
for  the  ROC,  TDR,  or  LR,  their  quantification 
wull  not  be  deferred  to  a later  time. 

(4)  A mandatory  RAM  rationale  annex  will 
be  prepared  for  each  ROC,  LR,  or  TDR  docu- 
ment to  justify  fully  the  RAM  category  and  sub- 
categories  that  are  used.  Quantitative  RAM  values 
will  be  justified  in  the  jointly  prepared  RAM 
rationale  annex.  For  ROC’s,  this  annex  will  be 
available  for  review  at  the  time  of  the  ROC  sub- 
mission to  I1QDA.  Justification  of  RAM  require- 
ments in  the  rationale  annex  will  include  and  be 
based  on — 

(а)  The  best  available  quantitative  and 
qualitative  estimates  of  the  operational  mode 
summary/mission  profile  (OMS/MP). 

(б)  Reliability  failure  definition  and,  when 
possible,  scoring  criteria  (FD/SC)  for  each  relia- 
bility requirement  used. 

(c)  Definitions  for  each  availability  and 
maintainability  parameter  used. 

(d)  Relevant  parts  of  the  Cost  and  Opera- 
tional Effectiveness  Analysis  (COEA)  which 
substantiate  the  RAM  requirements. 

(ct  A baseline  analysis  of  all  pert'nent  his- 
torical RAM  data. 

(/)  An  analy  sis  of  current  and/or  projected 
ceilings  of  RAM  feasibility  and  attainability  based 
on  state-of-the-art  considerations. 

(g)  An  analysis  of  the  RAM  impact  on 
system  effectiveness  and  logistic  supportability. 

(5)  L'nless  otherwise  qualified,  the  stated 
RAM  requirements  represent  values  which  apply 
to  functional  hardware  used  under  operational 
conditions.  As  such,  the  quantitative  values  will 
represent  the  hardware  in  an  operational  environ- 
ment (i.e.,  operated  in  accordance  with  the  OMS, 
MP)  from  DT  and/or  OT,  but  normally  these  re- 
quirements will  not  include  operator  or  mainte- 
nance errors. 
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(6)  RAM  characteristics  arc  measures  of 
technical  and  operational  hardware  performance 
and  logistic  supportability.  These  requirements 
are  based  on  need  and  realism  and  will  be  expressed 
only  in  terms  of  acceptable  values. 

(7)  Statements  such  as  “The  RAM  charac- 
teristics of  this  system  will  be  equal  to  or  better 
than  the  system  which  it  is  intended  to  replace” 
will  not  be  used.  Supportable  quantitative  require- 
ments will  be  listed  instead.  Additionally,  indi- 
vidual RAM  requirements  which  can  be  computed 
from  other  requirements  will  not  be  stated  since 
they  would  be  mathematically  redundant. 

2-7.  Cost  and  operational  effectiveness  analysis 
(COEA).  The  COE  A prepared  by  the  combat 
developer  will  quantitatively  consider  and  evaluate 
the  simultaneous  impact  that  RAM  has  on  both 
system  life  cycle  cost  and  operational  effectiveness. 
This  treatment  of  RAM  in  the  COEA  will  serve 
as  one  of  the  means  for  establishing  justifying  or 
validating  the  RAM  requirements. 

2-8.  Development  plan  (DP).  The  DP  (AR  70-27) 
is  the  controlling  document  for  the  materiel  de- 
velopment effort  and  will  contain  a complete  de- 
scription of  the  RAM  requirements  (sec  II  and 
III),  RAM  program  plan  and  activities  (sec  III), 
RAM  testing  (sec  IV),  RAM  resources  (sec  III 
and  IV),  and  critical  RAM  issues  (sec  I,  II,  III, 
IV,  VI).  The  DP  interfaces  with  logistics  planning 
(sec  VI)  and  system  engineering  (sec  III)  activities. 
2-9.  Key  management  documents.  The  key 
operational  hardware  RAM  requirements  will  be 
stated  in  Decision  Coordinating  Papers  (DCP’s), 
Defense  Program  Memorandums  (DPM’s)  and 
Army  Program  Memorandums  (APM’s).  Critical 
RAM  issues  to  be  addressed  during  development 
and  operational  test  and  evaluation  will  also  be 
included. 

2-10.  Solicitations  and  contracts,  a.  As  a result  of 
the  jointly  agreed  to  reliability  requirements 
stated  in  the  ROC,  Lll,  or  TDR,  the  materiel 
developer  will  conduct  a trade-off  analysis  to 
determine  the  specified  value  of  reliability  for 
contracting  purposes.  This  analysis  should  when 
applicable,  include  an  evaluation  of  changes  in 
costs  for  development  and  procurement  due  to 
incremental  changes  in  RAM  characteristics 
versus  the  associated  improvement  in  mission 
effectiveness  and/or  savings  in  projected  life 
cycle  cost.  This  trade-off  analysis  and  the  COEA 
will  be  closely  coordinated  efforts.  The  results  of 

TAGO  44A 


v*v. 


this  trade-off  analysis  will  be  used  by  the  materiel 
developer  in  determining  the  solicitation  and 
contract  specified  value  for  reliability  stated  in 
the  system  specification.  The  materiel  developer 
will  provide  the  proposed  specified  value  (SV ) for 
reliability  to  the  combat  developer.  The  SV  will 
be  used  by  the  developing  agency  as  the  design 
requirement  to  be  produced  and  delivered  by  the 
contractor(s).  The  contract  specified  reliability 
value  will  be  at  a level  necessary  to  insure  demon- 
stration of  the  reliability  requirements  stated  in 
the  requirements  document. 

b.  Solicitations  and  contracts  for  develop- 
ment of  materiel  will  include  requirements  in 
section  F for  RAM  programs  and  management 
controls,  in  addition  to  the  test  and  demonstra- 
tion provisions  and  the  quantitative  RAM 
requirements  in  the  contract  specifications.  These 
requirements  will  be  tailored  to  the  specific  life 
cycle  phase  of  the  system  being  developed. 

c.  RAM  incentives  and  award  fees,  will  be 
used  when  they  are  feasible,  advantageous  and 
beneficial  to  the  Army. 

d.  Contracts  for  the  procurement  of  materiel  will 
include  appropriate  requirements  to  insure  that 
the  materiel  being  procured  is  reliable,  durable, 
supportable,  and  maintainable. 

e.  Reliability  improvement  warranties  will  be 
used  when  it  is  feasible,  advantageous  and  bene- 
fical  to  the  Army. 

/.  The  capability  of  a contractor  to  define, 
design,  achieve  and  produce  materiel  which  meet 
the  stated  RAM  requirements  will  be  a major 
evaluation  factor  in  source  selection  (AR  715-6). 
When  data  are  available,  an  assessment  will  be 
made  and  utilized  in  source  selection  concerning 
the  contractor’s  ability  for  RAM  performance. 
The  solicitation  will  contain  appropriate  require- 
ments in  section  C for  the  RAM  content  of  the 
contractor’s  proposal.  As  necessary,  the  solicita- 
tion will  state  the  RAM  evaluation  factors  in 
section  D. 

g.  Technical  data  packages  (TDP)  will  be  de- 
veloped with  integral  RAM  functional  charac- 
teristics for  system  level  and  key  critical  lowe  r- 
level  work  breakdown  structure  elements.  The 
materiel  developer  will  place  appropriate  inspec- 
tion and  test  requirements  in  the  TD1’  to  injure 
the  delivery  of  a product  that  achieves  its  specified 
reliability  and  maintainability  requirements. 
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h.  When  appropriate,  the  materiel  developer 
will  develop  and  insert  RAM  requirements  to  the 
lowest  work  breakdown  structure  level  necessary 
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(e.g.,  3rd  or  4th)  in  order  to  control  the  RAM 
characteristics  to  be  specified  in  future  repair  part 
procurement  actions. 


Section  III.  RAM  CHARACTERISTICS 


2-11.  RAM  quantification,  a.  Reliability  require- 
ments will  normally  be  quantitatively  expressed 
as  the  attribute  mean-time  (or  distance,  or 
rounds) — between  failure  (MTBF,  MDBF,  or 
MRBF)  except  when  such  parameters  as  mission 
reliability,  or  in-flight  reliability  are  more  ap- 
propriate. MTBF,  MDBF,  MRBF,  or  mission 
parameters  trill  be  associated  with  the  OMS/MP 
and  will  be  accompanied  by  a FD/SC  which  de- 
scribe these  reliability  parameters  in  terms  of 
chargeable  failures.  Additional  reliability  param- 
eters (such  as,  mean-time-bet  ween-unscheduled- 
maintenance-(MTBUMA)  or  mean-time-between- 
any-maintenance-action  (MTBAMAj)  will  be 
stated  when  appropriate  and  these  parameters 
trill  be  associated  with  the  overall  maintenance 
burden  of  the  system.  Unless  otherwise  stated,  all 
reliability  requirements  will  pertain  to  new  equip- 
ment. When  appropriate,  additional  reliability 
requirements  for  rebuilt  or  overhauled  equipment 
may  be  established  and  will  be  clearly  labeled. 

b.  Determination  of  the  reliability  requirement 
will  be  jointly  developed  by  a combat  and  ma- 
teriel developer  working  group  and  must  be  agreed 
to  by  both  the  combat  and  materiel  developers.  The 
reliability  requirement  will  be  stated  as  both  the 
minimum  acceptable  value  (MAV)  and  the  best 
operational  capability  (BOC).  The  MAV  for 
reliability  is  defined  as  that  value  which  presents 
the  least  operational  capability  the  user  can 
tolerate  aud  accept.  It  represents  a level  of  mar- 
ginal performance  which  is  the  floor  value  of 
acceptability  and  below  which  is  unacceptable. 
During  DT/OT  testing  which  immediately  pre- 
cede- the  type  classification  (standard)  decision, 
the  MAV  will  have  a low  consumer’s  risk  associ- 
ated with  it.  It  will  normally  be  based  on:  The 
user’s  minimum  requirements  for  mission  success; 
the  logistician’s  minimum  requirements  to  provide 
responsive,  economical  logistic  support;  and  man- 
datory requirements  of  national  or  international 
laws  and  agreements,  such  as  environmental 
control  and  nuclear  safety.  While  the  MAV 
represents  the  floor  of  acceptability,  the  user’s 
prefm  nco  for  reliability  increases  as  reliability 
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increases  beyond  the  MAV.  The  BOC  value  for 
reliability  is  defined  as  that  upper  level  of  relia- 
bility which  is  estimated  to  be  technically  feasible 
for  the  stated  time  frame  within  reasonable  cost 
constraints  and  is  in  consonance  with  the  highest 
reliability  performance  for  which  a realistic  need 
exists. 

c.  Reliability  after  storage  may  be  a stated 
requirement.  If  appropriate,  it  will  specify  the 
amount  of  deterioration  acceptable  during  storage. 
Length  of  storage,  storage  environment,  and 
surveillance  constraints  will  be  identified  (AR 
70-38).  It  should  be  recognized  that  this  require- 
ment may  not  be  testable  and  may  rely  on  an 
engineering  analysis  for  its  assessment  prior  to 
deployment. 

d.  Durability  requirements  may  be  stated  for 
systems  and/or  major  components  (e.g.,  engine, 
transmission,  gun  tube),  and  will  use  the  most 
appropriate  quantitative  parameter(s)  and  dura- 
bility failure  definition.  Durability  will  be  periodi- 
cally' assessed  throughout  the  life  cycle.  Durability 
requirements  will  be  developed  to  optimize,  over- 
haul or  rebuild  policies  and  to  minimize  cost.. 

c.  Availability  will  be  expressed  as  operational 
availability  (Ao),  achieved  availability  (Aa), 
inherent  availability  (Ai)  or  combinations  thereof. 
Definitions  for  Ao,  Aa,  and  Ai  are  in  Appendix  A. 

j.  When  Ao  is  quantified,  *he  following  apply: 
(1)  While  Ao  parallels  the  intent  of  equip- 
ment operationally  ready  rates  (AR  750-52),  Ao 
is  usually  not  synonomous  with  operational 
readiness.  Ao  represents  a number  used  in  the 
development  phase  to  convey  an  approximate 
estimate  of  future  equipment  operationally  ready 
rates.  For  the  two  to  be  directly  comparable, 
FD  and  SC  used  in  testing  must  be  the  same  as 
Equipment  Serviceability  Criteria  ESC)  list'd 
in  operational  units  to  derive  operationally  ready 
rates,  actual  equipment  usage  in  the  field  must 
be  essentially  the  same  as  the  tested  OMS  MP,  and 
logistic  support  provided  in  the  field  must  be 
similar  to  the  logistic  support  simulated  in 
operational  testing.  Due  to  the,  inclusion  of  both 
operating  time  and  standby  time,  any  use  of  Ao 
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must  always  be  accompanied  by  an  item  utilization  will  state  whether  it  is  computed  from  system 

rate  (i.e.,  amount  of  required  operating  time  per  operating  hours  or  engine  hours. 

-ome  calendar  time  period).  In  a similar  fashion,  j.  Availability,  maintainability  and  durability 
Ao  must  be  factorable  into  its  operational  hard-  requirements  will  normally  be  expressed  as  single 

ware  reliability  and  maintainability  characteristics  values  and  will  be  used  by  the  materiel  developer 

along  with  total  administrative  and  logistics  as  minimum  design  requirements  to  be  produced 

downtime  information  in  order  to  influence  the  and  delivered  by  the  contractor(s)  and  demon- 

hardware  design  and  the  design  of  the  logistics  strated  in  tests. 

system  and  the  maintenance  support  concept.  h.  From  a RAM  and  logistic  viewpoint,  it  i- 

(2)  The  rationale,  assumptions,  and  data  used  sometimes  equally,  if  not  more  important,  to 

to  establish  Ao  criteria  will  be  stated  in  relation  quantify  maintenance  burden  requirements  in 

to  the  mission  duration  or  profile  so  that  proper  addition  to  or  in  place  of  operational  mi— ion 

trade-oil's  can  be  made  among  reliability  and  main-  requirements.  In  these  ca-cs  an  annual  support 

taxability  characteristics  with  operational  require-  cost  (see  appendix  A)  characteristics  may  be 

ments  and  logistical  support  considerations.  The  developed.  ASC  would  include  the  direct  support 

description  of  the  Ao  characteristic  must  be  costs  (i.e.,  personnel  and  parts)  attributable  to 

explicit  (i.e.,  available  to  do  what,  and  under  what  ami  controlled  by  the  design, 

operating  and  support  environment)  and  will  I.  As  needed,  a parameter  to  quantify  “off- 
clearly  define  how  to  factor  the  total  calendar  system”  maintenance  actions  may  be  developed 

time  and  how  to  charge  incidents  or  system  events  and  stated. 

to  the  Ao  formula  categories.  2 12.  Operational  mode  summary  and  mission 

(j.  Mean-time-to-repair  (MTTR)  may  be  used  profile  ( OMS/MP).  a.  The  operational  mode 

to  quantify  the  system’s  maintainability  char-  summary  describes  the  anticipated  mix  of  opera- 

acteristic.  MTTR  applies  to  the  configuration  tional  modes  and  their  expected  percentages  of 

system  level  and  will  be  used  as  an  “on-system”  use  in  different  operational  and  environmental 

maintainability  index  and  not  for  the  repair  of  conditions  for  winch  the  system  was  designed 

components.  It  does  not  include  off-system  repairs  throughout  its  life  before  overhaul.  The  mission 

of  components  that  are  replaced.  The  MTTR  will  profile  describes  the  operational  requirement (s) 

be  stated  for  both  organization  and  direct  support  that  a system  must  complete  to  accomplish  a 

levels  of  maintenance.  particular  mission  or  set  of  missions  (i.e.,  battle- 

h.  Maximum  time  to  repair  or  maximum  cor-  field  day).  The  operational  mode  summary  will 

rective  maintenance  downtime  may  be  used,  if  be  based  on  the  specific  mission  profile(s)  that  the 

appropriate,  to  quantify  maintainability.  It  will  user  estimates  to  be  the  essential  specific  task-  the 

not  include  off-system  repairs  of  replaced  equipment  will  be  required  to  perform  and  that 

components.  portion  of  the  life  expected  to  be  spent  in  these 

i.  Maintenance  ratio  (MR)  will  be  a measure  of  missions  or  other  use. 

the  total  maintenance  manpower  burden  required  b.  The  operational  mode  summary  will  be  used 
to  maintain  a system.  It  is  expressed  as  the  cumu-  to  determine  the  RAM  requirements  and  overall 

lative  number  of  man-hours  of  maintenance  ex-  combat  effectiveness.  As  necessary,  mission  re- 
pended in  direct  labor  during  a given  period  of  liability  will  be  determined  from  the  mission 

time,  divided  by  the  cumulative  number  of  end  profile.  Evaluation  of  life  cycle  costs  in  the  COEA 

item  operating  hours  (or  rounds  or  miles)  during  will  include  other  anticipated  use  modes  such  as 

the  same  time.  The  maintenance  ratio  will  be  training.  The  test  program  will  be  designed  to 

expressed  for  specific  levels  of  maintenance  and  provide  data  for  evaluation  of  the  required 

summarized  for  combined  levels  of  maintenance.  characteristics  in  terms  of  the  operational  mode 

All  maintenance  actions  are  considered  (i.e.,  summan  and  also  for  evaluation  of  the  specific 

scheduled  as  well  as  corrective,  and  without  regard  mission  profile(.-i  in  relation  to  age  and  wear  of 

to  their  effect  on  availability  of  the  system).  Man-  the  system. 

hours  for  off-system  repair  of  replaced  components  c.  The  operational  mode  summary  again-t  which 
or  daily  operational  checks  are  not  included.  For  government  testing  (i.e.,  i)T,  O F,  or  both  DT 

items  like  combat  vehicles,  the  MR  requirement  and  Old  and  evaluation  are  performed  will  be 
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representative  of  actual  operational  requirements. 
RAM  characteristics  will  be  evaluated  in  accord- 
ance with  the  relative  frequency  of  uses  defined  in 
the  operational  mode  summary  rather  than  over- 
all inclusive  potential  uses  or  at  the  rare  extreme 
uses. 

d.  Operational  mode  summary/mission  profde 
will  be  developed  by  the  combat  developer  and 
coordinated  with  the  materiel  developer,  the 
operational  tester  and  development  tester  for 
review  and  comment.  The  OMS/MP  will  be 
developed  prior  to  the  submission  of  a proposed 
ROC  to  IIQDA.  The  OMS/MP  will  be  used  as 
the  basis  for  determining  and  including  RAM 
parameters  in  the  ROC,  LR,  or  TDR.  The 
OMS/MP  will  be  included  in  the  RAM  rationale 
annex,  DP,  and  CTP. 

2-13.  Failure  definition  and  scoring  criteria,  a.  For 

a reliability  parameter  like  MTBF  to  convey 
meaning,  an  accompanying  definition  must  be 
established  to  define  what  constitutes  a failure 
and  what  events  may  be  excluded.  A failure 
definition  consistent  with  the  OMS/MP  will 
be  developed  and  stated  in  the  requirements 
document  or  rationale  annex  thereto.  The  detailed 
scoring  criteria  will  amplify  the  failure  definition 
and  will  include  procedures  for  classifying  main- 
tainability, availability  and  durability  parameters. 
Detailed  scoring  criteria  will  be  developed  in  a 
time  phased  sequence  with  the  design  evolution 
process.  Both  the  failure  definition  and  the 
detailed  scoring  criteria  will  be  included  in  the 
DP,  CTP,  solicitations  and  contracts,  and  all 
other  appropriate  documents. 

6.  The  combat  developer  and  materiel  developer 
will  jointly  develop  and  formally  approve  the 
FD/SC  which  will  be  used  throughout  the  materiel 
life  cycle  process.  The  operational  and  develop- 
ment testers  and  evaluators  and  logistician  will 
review  and  comment  on  the  FD/SC  to  insure  that 
the  FD/SC  can  be  applied  during  actual  testing 
and  will  address  the  feasibility  of  collecting  the 
necessary  data.  The  FD/SC  must  be  approved 
prior  to  the  commencement  of  full  scale  engineering 
development.  Formal  agreements  on  FD/SC 

Section  IV. 

2-14.  Development  testing  (DT).  a.  DT  is  that 
testing  and  evaluation  conducted  to  demonstrate, 
evaluate,  or  assess  that  the  engineering  design 
and  development  process  is  complete,  that  the 
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will  be  included  in  either  the  normal  requirements 
or  program  documents  or  as  separate  agreements. 

c.  The  materiel  and  combat  developers  will 
develop  generalized  failure  definitions  for  each 
individual  category  of  equipment  (e.g.,  combat 
vehicles,  communications,  air  defense  missiles). 
Individual  scoring  criteria  will  be  developed  for 
specific  systems  within  each  category. 

d.  In  order  to  evaluate  both  reliability  and  the 
logistic  burden,  the  FD/SC  will  normally  provide 
for  such  incident  classification  categories  as: 

(1)  No  test — incidents  which  are  not  counted 
as  maintenance  actions  or  as  failures. 

(2)  No  failure — incidents  requiring  unsched- 
uled maintenance  but  which  are  not  failures. 

(3)  Failures. 

(а)  Incidents  which  require  unscheduled 
maintenances,  and  are  system  failures  but  not 
mission  abort  failures. 

(б)  Incidents  which  require  unscheduled 
maintenance  and  are  both  system  and  mission 
abort  failures. 

(c)  Incidents  which  require  no  maintenance 
action,  and  are  system  failures  but  not  mission 
abort  failures. 

(d)  Incidents  which  require  no  maintenance 
action,  and  are  both  system  and  mission  abort 
failures. 

e.  The  approved  FD/SC  will  be  used  through- 
out the  materiel  life  cycle  to  consistently  assess 
the  RAM  requirements. 

/.  The  reliability  FD/SC  will  clearly  state  the 
methodology  and  procedures  to  be  invoked  in 
purging  obsolete  failures.  Obsolete  failures  are 
defined  as  patterned  failures  or  failure  modes 
which  as  a result  of  a design  change  and  subse- 
quent verification  throughout  are  not  expected 
to  recur.  In  purging  failures  on  development 
systems,  considerations  must  be  given  to  the 
effectiveness  of  the  design  fix  and  the  fix  verifi- 
cation through  test. 

g.  All  FD/SC  jointly  approved  prior  to  the 
effective  date  of  this  regulation  will  continue 
in  use.  Specific  changes  directed  towards  incor- 
porating the  above  FD/SC  policies  are  optional. 

TESTING 

design  risks  have  been  minimized,  and  that  the 
system  will  meet  specifications;  and  to  estimate 
the  system’s  military  utility  when  it  is  introduced. 
DT  is  accomplished  in  factory,  laboratory,  and 
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proving  ground  environments  and  includes  en- 
gineering design  testing  and  human  factors 
testing  to  demonstrate  a satisfactory  technical 
man  (soldier) -machine  interface  using  qualified 
and  experienced  operators,  crews,  and  mainte- 
nance support  personnel. 

b.  DT  will  be  designed  to  provide  a measure  of 
the  system  RAM  characteristic  against  stated 
requirements  in  a controlled  environment  with 
the  procedures  and  resources  contained  and/or 
described  in  the  Maintenance  Test  Support 
Package  (AR  750-1).  The  DT  RAM  emphasis 
will  be  to  identify  design  deficiencies  and  to  pro- 
vide valid  estimates  of  the  inherent  RAM  char- 
acteristics of  the  materiel  for  the  purpose  of 
providing: 

(1)  Assessment  of  RAM  growth. 

(2)  A basis  for  assessing  the  consequences  of 
any  differences  anticipated  during  field  operations. 

(3)  A basis  for  resolution  of  legal  (contrac- 
tual) issues  between  the  government  and  its 
contractor. 

(4)  Contribute  data  to  the  DT/OT  RAM 
data  base  for  aggregation  and  ROC  requirements 
comparison  (where  aggregation  is  statistically 
valid). 

(5)  A basis  for  a clear  identification  and 
understanding  of  both  failures  and  maintain- 
ability design  deficiencies  and  the  basis  for  taking 
cor-  ctive  measures. 

c.  Integrated  development  testing  for  reliability 
will  be  designed  and  conducted  to  duplicate  as 
closely  as  possible  the  anticipated  field  environ- 
mental profile.  Tailored  environmental  profiles 
should  be  developed  and  utilized.  Standard 
environmental  profiles,  like  those  contained  in 
Mil. -STD-781,  should  only  be  used  when  they 
are  the  “best  fit”  or  when  sufficient  environment 
information  is  not  available  and  cannot  be 
generated. 

2-15.  Operational  testing  (OT).  a.  OT  is  testing 
that  is  conducted  to  estimate  a prospective 
systems  military  utility,  operational  effectiveness, 
operational  suitability  (including  compatibility, 
interoperability,  RAM,  logistic  supportability, 
operational  man  (soldier)-machine  interface,  and 
training  requirements),  and  the  need  for  any 
modifications.  In  addition,  OT  provides  informa- 
tion on  organization,  personnel  requirements, 
doctrine  and  tactics.  It  may  also  provide  data  to 
support  or  verify  operating  instructions,  publica- 
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tions,  software  and  handbooks.  OT  will  be  ac- 
complished by  operational  and  support  personnel 
of  the  type  and  qualifications  of  those  who  are 
expected  to  use  and  maintain  the  system  when 
deployed. 

6.  OT  will  be  designed  to  evaluate  the  RAM 
performance  characteristics  upon  exposure  of  the 
materiel  to  a variety  of  expected  operational 
conditions.  OT  concentrates  on  RAM  performance 
when  in  the  hands  of  typical  user  troops  under 
operational  conditions.  OT  provides  data  for  the 
estimation  of  life  cycle  costs  and  logistic  resource 
requirements  and  also  provides  a basis  for  improv- 
ing the  nonmateriel  aspects  of  RAM  performance. 
OT  RAM  emphasis  will  be  to: 

(1)  Provide  valid  operational  estimates  of 
RAM  characteristics. 

(2)  Contribute  data  to  the  DT/OT  RAM 
data  base  for  aggregation  and  ROC  requirements 
comparison  (where  aggregation  is  statistically 
valid). 

(3)  Complement  DT  data  needs  by  instru- 
menting item/system  undergoing  OT,  recording 
detailed  failure  data,  and  allowing  failure  diagnosis 
by  materiel  developer  personnel  where  their 
efforts  do  not  detract  from  the  validity  of  the  OT. 

(4)  Determine  the  interaction  of  RAM 
characteristics,  including  system,  crew  and  logis- 
tical support  interfaces,  with  overall  -ystem 
performance. 

(5)  Represent,  to  the  maximum  degree 
possible,  realistic  operational  conditions  based 
on  the  jointly  agreed  to  operational  mode  sum- 
mary; mission  profile. 

2-16.  Test  documentation.  The  operational  tester 
and/or  evaluator  and  the  development  tester 
and/or  evaluator  will  produce  the  following 
documentation  which  will  describe  the  complete 
RAM  test  and  evaluation  plan  for  each  major 
phase  of  testing  (DT  and  OT  I,  II,  and  III  or 
follow-on  operational  test  and  evaluation 
(FOTE)).  (Reference  AR  70-10.) 

a.  Independent  evaluation  plan  ( 1EP ).  The 
evaluation  methodology  and  criteria  to  be  em- 
ployed in  the  evaluator’s  independent  evaluation 
with  respect  to  RAM  characteristics  and  logistics 
supportability  will  be  included  in  the  IEP. 

b.  Test  design  plan  (TDP).  All  RAM  testing, 
whether  factory,  laboratory,  or  proving  ground 
(for  DT)  or  operational  unit  (for  OT)  will  be 
described  and  related  to  RAM  critical  issues,  if 
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uny.  RAM  test  plans  will  implement  t he  te~t 
design  plan. 

r.  Detailed  tent  plans  (DTP).  These  are  the 
tester's  internal  procedures  for  executing  the  tests 
per  the  TDP.  RAM  test  plans  will  describe  the 
data  storage  and  retrival  system,  the  statistical 
and  engineering  techniques  to  be  used  in  data 
analysis  and  the  type  of  quantitative  and  quali- 
tative results  to  be  produced. 

d.  Test  reports.  The  test  unit  or  agency  assigned 
responsibility  to  conduct  tests  contributing  to  the 
overall  evaluation  will  report  results  of  RAM  tests 
and  demonstrations  in  detail,  complete  with  report 
or  estimate  of  the  cause  of  the  failure  and  the 
manner  in  which  the  failure  occurred,  and  discus- 
sions of  the  importance  and  impact  of  defects 
found  in  system  RAM. 

e.  Independent  evaluation  reports  (IER).  The 
assigned  evaluators  for  OT  and  for  DT  will  present 
a complete  evaluation  of  demonstrated  or  predicted 
RAM  characteristics  and  their  assessment  of  the 
military  utility  or  system  effectiveness  resulting 
therefrom.  Departures  from  the  results  of  scoring 
conferences  will  be  identified  for  the  decision- 
making body  (IPR.  ASARC,  or  DSA RC). 

2-17.  Test  planning  and  design,  a.  It  is  expected 
the  DT/OT  I RAM  data  will  be  primarily  tech- 
nical in  nature  but  some  limited  operational  RAM 
data  may  be  obtained  from  this  testing. 

b.  DT  OT  IT  proceeds  the  first  production  deci- 
sion. Prior  to  this  decision,  it  is  necessary  to  have 
valid  estimates  of  the  technical  and  operational 
RAM  characteristics.  Adequate  testing  must  be 
performed  in  both  DT  II  and  OT  II  to  provide 
these  estimates.  Aggregation  of  data,  when  pos- 
sible. will  improve  the  overall  estimates,  but  nor- 
mally. RAM  results  from  either  DT  IT  or  OT  II 
will  stand  alone.  Valid  estimates  of  RAM  char- 
acteristics from  an  operational  environment  are 
essential.  However,  the  independent  objectives 
of  DT  II  and  OT  II  must  be  maintained  and  not 
sacrificed  at  the  expense  of  one  test  over  the  other. 

c.  Data  from  the  DT/OT  III  test  phase  must 
support  the  decision  as  to  whether  the  system 
should  be  deployed.  Valid  estimates  of  RAM 
characteristics  from  an  operational  environment 
are  essential.  Data  will  be  aggregated  when 
possible. 

(/.  >tati-tically  designed  experiments  will  be 
used  to  the  maximum  extent  pos-ible  in  planning 
the  Coordinated  Test  Program  to  assure  that 
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optimum  information  is  obtained  from  the  testing 
effort  commensurate  with  the  effective  utilization 
of  funds,  resources,  and  time  during  development. 

e.  DT  and  OT  will  be  designed  to  provide 
sufficient  data  to  permit  identification  of  trends 
in  equipment  failure  rates  (i.e.,  increasing,  con- 
stant, decreasing)  and  the  determination  of 
applicable  statistical  distribution  for  reliability, 
availability  or  maintainability  analysis.  Reli- 
ability testing  will  allow  assessment  throughout 
the  life  cycle  before  overhaul,  rebuild,  or  service 
life,  whichever  is  appropriate.  DT  will  be  designed 
to  identify  failure  modes  and  design  deficiences 
in  order  that  corrective  action  may  be  taken  to 
promote  reliability  growth  to  the  specified  value. 

f.  Test  planning  and  the  design  of  DT  and 
OT,  will  recognize  that  DT  and  OT  RAM  data 
will  normally  yield  three  different  types  of  RAM 
indices:  DT  RAM  indices,  OT  RAM  indices, 
and  the  aggregated  DT/OT  RAM  indices.  The 
DT  RAM  indices  are  inherent  technical  hardware 
values.  The  OT  RAM  indices  are  overall  system 
values  which  include  failures  of  the  human  element 
and  all  other  field  operational  factors.  The 
aggregated  RAM  indices  are  derived  from  com- 
bined DT  OT  test  data  which  are  processed 
through  the  FI)  SO  and  is  representative  of  future 
field  use  and  which,  when  the  data  can  be  validly 
combined,  will  be  used  to  assess  the  compliance 
with  ROO  or  LR  RAM  requirements.  RAM  data 
aggregation  will  consider  that  the  stated  RAM 
requirements  are  normally  intended  to  represent 
the  technical  hardware  values  when  the  hardware 
is  operated  in  accordance  with  the  OMS/MP. 
Tn  cases  where  the  RAM  results  from  DT  and 
from  OT  are  expected  to  be  different,  data 
aggregation  may  not  be  feasible.  In  these  eases, 
test  plans  should  be  structured  to  document  the 
reasons  for  any  observed  differences  in  system 
performance.  Differences  in  RAM  values  that 
may  occur  between  DT  and  OT  results  are  normal 
and  their  occurrence  should  not  be  misconstrued 
as  unfavorable  RAM  indicators.  Reasons  for 
these  differences  may  include  different  environ- 
ments, man-machine  interfaces,  test  designs, 
level  of  training,  or  differences  in  the  precision 
of  data  collection  and  subsequent  scoring  in  DT 
and  < >T. 

a.  Test  planning  will  be  such  that  all  RAM 
data  from  DT  and  < tT  can  be  provided  and 
exchanged  among  all  principal  parties  in  the 
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materiel  acquisition  process,  but  particularly 
the  combat  developer,  materiel  developer,  develop- 
ment testers/evaluators,  operational  testers/ evalu- 
ators and  logistician  on  a timely  and  responsive 
basis.  Data  collection  requirements  will  be  co- 
ordinated among  the  parties  in  sufficient  time 
before  te't  initiation  to  insure  that  all  data  needs 
are  fulfilled. 

/(.  Preparation  of  the  coordinated  test  program 
(CTP)  (DA  Pam  70-21)  requires  analysis  to 
verify  RAM  testing  is  adequate  to — 

(1)  Detect  patterned  failures. 

(2)  Permit  planned  reliability  growth. 

(3)  Provide  an  acceptable,  and  consistent 
basis  for  determining  the  systems  RAM  status  for 
presentation  to  the  decision  body. 

(4)  Insure  that  both  user’s  and  producer’s 
risks  are  adequately  addressed. 

Appropriate  mathematical  techniques  will  be 
employed  in  test  design  to  accomplish  these  goals. 

i.  When  using  a quantified  analysis  of  testing 
risk  to  assist  in  test  design,  the  risk  levels  for  DT 
and  OT  will  be  established  based  on  the  criticality 
and  user  impact  of  reliability  and  they  will  be 
documented  in  the  CTP. 

j.  Any  quantification  of  test  risk  will  also  in- 
clude a sensitivity  analysis.  This  total  analysis  in 
conjunction  with  engineering  judgment  and  re- 
source constraints  will  be  used  to  develop  the 
recommended  test  length. 

k.  Care  must  be  taken  to  maintain  a balance 
among  DT  II,  OT  II,  DT  III,  OT  III,  and  FOTE 
which  are  commensurate  with  test  objectives  and 
acquisition  strategy.  From  a production  decision, 
operational  suitability  and  reliability  growth 
viewpoints,  the  combined  DT/OT  II  should  be 
of  greater  importance  than  the  combined  DT/OT 
III.  DT  II  should  not  be  sacrificed  for  OT  II. 

l.  Maximum  use  will  be  made  of  military  main- 
tenance personnel  during  early  testing  and 
mock-up  evaluations.  It  is  during  this  period  that 
the  greatest  savings  can  be  realized  in  the  time 
and  resources  which  will  ultimately  be  required 
to  maintain  t lie  equipment.  The  materiel  developer 
should  provide  for  observation/participation  by 
military  personnel  in  such  areas  as  contractor 
testing,  mock-up  reviews  and  early  maintain- 
ability evaluations. 

T".  Test  plans  for  each  phase  of  the  development 
cycle  should  be  designed  based  on  the  requirement 
and  tin  planned  leliabilily  growth  and  will  be 

TACJO  44 A 


AR  702-3 

different  for  engineering  development  (DT/OT  II) 
and  initial  production  (DT/OT  III).  Early 
development  and  operational  test  DT/OT  I and 
DT/OT  II  of  a system  occurs  at  a point  on  the 
reliability  growth  curve,  that  is  normally  less  than 
the  specified  value.  DT  I/OT  I and  DT  II/OT  II 
should  be  designed  to  evaluate  RAM  characteris- 
tics anti  assure  program  continuation  if  sufficient 
reliability  growth  has  been  achieved.  In  general, 
sufficient  reliability  growth  will  have  been  achieved 
if  the  MAV  is  demonstrated  at  high  confidence 
during  the  DT/OT  preceding  the  decision  to  type 
classify  standard.  Improvement  of  reliability  will 
be  evaluated  against  the  requirement  and  the 
projected  value  on  the  reliability  growth  curve 
for  the  appropriate  point  in  the  development  or 
production  phase. 

7i.  OT  II,  OT  III  and  FOTE  will  be  designed 
to  assess  the  logistic  considerations  of  the  system 
in  conjunction  with  operational  availability. 
At  OT  II  the  logistic  concept  will  be  reviewed  and 
implemented  where  possible.  The  review,  analysis, 
and  evaluation  of  test  results  will  identify  potential 
logistic  problems.  At  OT  III  or  in  FOTE,  the 
logistics  concept  will  be  implemented  to  the 
maximum  extent  to  allow  an  assessment  of  the 
concept  on  the  system’s  operational  effectiveness. 

o.  Normally  OT  II  will  address  maintenance 
through  general  support  level  using  trained 
military  personnel.  Characteristically,  OT  III 
will  exercise  maintenance  support  throughout 
GS  level,  with  trained  military  personnel  required 
through  that  level.  In  the  case  of  tests  where 
waivers  have  been  granted  to  go  into  tests  without 
the  full  logistic  system  planned,  assessment  of 
maintenance  may  be  limited  with  the  degree  of 
assessment  determined  by  the  operational  tester 
or  a case  by  case  basis. 

p.  Normally  OT  will  not  assess  durability,  but 
will  contribute  data  to  the  developer  for  that 
assessment.  The  nature  of  durability  is  such  that 
testing  for  it  necessitates  a technical  assessment. 
A test  item  may  be  used  in  both  DT  anil  OT  to 
accumulate  the  necessary  operating  time  equiv- 
alent to  its  useful  life  for  the  assessment  of  durabil- 
ity. The  operational  tester  will  not  attempt  to 
apply  a statistical  significance  to  any  such  failures 
which  occur,  but  such  failures  will  be  considered 
in  evaluation. 

q.  The  use  of  point  estimate  test  criteria 
(para  2 -19)  should  not  be  construed  as  permitting 


»*5‘,  •" 


2-1 1 


AR  702-3 


15  November  1976 


the  length  of  the  test  to  vary  without  bounds.  to  a full  production  decision.  The  criteria  are  not 
Tests  should  be  designed  such  that  the  test  length  intended  to  be  all  inclusive  since  issues  unforeseen 

is  of  adequate  and  sufficient  length  to  detect  at  planning,  often  arise  as  the  result  of  testing, 

patterned  failures,  to  permit  reliability  growth,  The  test  criteria  should  reflect  the  projected  growth 

and  to  control  to  a tolerable  level  the  subsequent  of  system  performance  during  the  development  cycle, 

estimation  variability  in  data  analysis.  b.  Normally  the  reliability  test  criteria  will  be  as 

2-18.  Test  criteria,  a.  In  accordance  with  AR  follows: 

70  10,  intra-Army  reliability  test  criteria  will  be  (1)  When  the  material  acquisition  plan  for  a 

jointly  defined  by  the  materiel  and  combat  particular  system  calls  for  a low'  rate  initial  pro- 
developers for  each  phase  of  acquisition  testing.  duction  (LRIP)  decision  followed  by  a period  of 

Test  criteria  will  be  coordinated  with  the  opera-  testing  and  RAM  growth  prior  to  the  full  produe- 

tional  and  development  testers  and  included  in  tion  decision,  a point  estimate  value  of  the  MAV 
the  CTP  prior  to  its  coordination  and  approval  will  be  achieved  under  operational  conditions 

for  the  initial  phase  of  testing  and  recoordinated  prior  to  the  LRIP  decision. 

during  CTP  update.  These  criteria  will  be  es-  (2)  The  MAV  at  high  confidence  will  be 

tablished  using  the  reliability  requirements,  re-  demonstrated  under  operational  conditions  in  DT/ 

liability  growth  considerations,  and  technological  OT  prior  to  a full  production  decision, 

assessment  of  the  development  program.  Since  no  c.  Availability  and  maintainability  test  criteria 
single  test  or  series  of  tests  can  provide  all  the  for  Di'/OT  prior  to  any  type  classification  de- 
information upon  which  to  base  a decision,  these  cision  will  be  a point  estimate  which  is  equal  to  or 
reliability  criteria  are  not  established  as  automatic  better  than  the  stated  requirements. 

“pass/fail”  criteria  for  the  system,  but  are  criteria  d.  The  above  test  criteria  are  guidelines.  Indi- 
to  measure  the  attainment  of  the  reliability  re-  vidual  test  criteria  must  be  tailored  for  each 

quirement.  The  test  criteria  will  be  used  to  assess  system  and  reflect  both  the  stated  RAM  require- 

satisfactory  reliability  progress  in  early  DT  and  ments,  the  impact  on  system  performance  of 

03'  and  to  assess  the  demonstrated  achievement  achieving  various  RAM  levels  and  the  available 

of  the  reliability  requirement  at  the  D'I'/OT  prior  testing  resources. 

Section  V.  ASSESSMENTS 

2-19.  Scoring  conferences,  a.  Formal  scoring  against  RAM  requirements.  To  accomplish  this 

conferences  between  the  materiel  developer  pro-  objective,  scoring  conferences  must  jointly  estab- 

ponent,  combat  developer  proponent,  (preferably  lish  the  facts  surrounding  test  incidents,  determine 

the  same  representatives  who  established  the  if  the  facts  warrant  classifying  the  test  incidents 

FD/SC)  operational  tester,  operational  evaluator,  as  chargeable  failures  in  accordance  with  the 

development  tester,  and  the  development  evalua-  scoring  criteria,  and  agree  upon  the  test  incident 

tor  will  be  held  during  and/or  immediately  after  facts  and  classifications.  The  result  of  the  process 

the  conduct  of  a test  to  assure  that  a proper  and  is  a single  test  data  base  identifying  test  length 

consistent  determination  is  made,  for  categorizing  (hours,  distance,  or  rounds)  and  chargeable 

test  data  to  permit  the  assessment  of  the  relia-  incidents  which  all  parties  agree  to  use  in  assessing 

bilitv,  availability,  and  maintainability  ROC'  the  achievement  of  RAM  requirements.  However, 

requirements.  Results  (includes  incident  charge-  a clear  distinction  must  be  made  between  agreeing 

ability  and  minority  opinions)  of  the  scoring  on  the  test  data  base  (the  facts)  and  the  analysis 

conferences  will  be  included  in  the  applicable  of  that  data  base  (the  interpretation  of  the  facts), 

test  report.  Minutes  of  the  scoring  conferences  Subsequent  independent  evaluations  or  assess- 

wrill  be  provided  to  the  attendees  and  designated  ments  will  address  the  achievement  of  RAM  or 

logistician.  the  impact  of  what  happened  during  test.  The 

b.  The  objective  of  the  scoring  conference  is  to  independent  evaluators  may  analyze  the  test 

establish  a test  data  base  to  assure  that  a proper  data  base  in  any  appropriate  manner  as  long  as 

and  consistent  determination  is  made  for  cate-  any  deviations  from  the  agreed  upon  classification 

gorizing  test  incidents  as  failures  chargeable  are  clearly  identified. 
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c.  The  materiel  developer  proponent,  combat 
developer  proponent,  operational  evaluator,  ami 
development  evaluator  will  each  designate  a 
single  spokesman  prior  to  the  scoring  conference. 
These  spokesmen  must  have  the  authority  to 
speak  for  their  Agency.  Decisions  on  chargeable 
failures  will  he  through  the  unanimous  consensus 
of  the  principal  spokesman.  If  unresolved  differ- 
ences do  exist,  the  differing  opinion(s)  will  be 
formally  documented  in  the  minutes  of  the  meet- 
ing. The  incident  chargeability  for  the  assessment 
of  RAM  requirements  will  then  be  based  on  the 
majority  opinion.  In  those  cases  where  there  is  no 
majority  opinion,  the  combat  developer  propo- 
nent will  make  the  final  determination  of  incident 
chargeability. 

d.  The  materiel  developer  proponent  will 
chair  all  DT  scoring  conferences,  while  the  opera- 
tional evaluator  will  chair  all  OT  scoring  con- 
ferences. The  evolution  of  the  DT,  OT  and,  if 
possible,  aggregated  DT/OT  numbers  will  be 
through  these  scoring  conferences.  At  the  com- 
pletion of  OT,  there  will  be  a final  conference, 
chaired  by  the  operational  evaluator  for  the 
purpose  of  reviewing  all  DT  and  OT  RAM  data, 
establishing  a valid  aggregated  data  base,  and 
developing  the  aggregated  RAM  numbers. 

e.  The  operational  evaluator,  developer  eval- 
uator, combat  developer  and  materiel  developer 
will  determine  what  data  may  be  aggregated. 
Two  factors  will  be  considered  in  determining  if 
data  can  be  aggregated.  First,  data  from  DT  to 
be  aggregated  with  OT  data  must  come  from 
testing  that  was  conducted  in  accordance  (approx- 
imate) with  the  OMS/MP.  That  is,  conditions 
must  be  operational  and  include  utilization  rates 
in  accord  with  the  OMS  and  maintenance  pro- 
cedures. If  it  is  agreed  such  conditions  did  exist 
in  DT  and/or  OT  then  the  second  factor,  that  of 
valid  statistical  data  correlation  must  be  con- 
sidered. The  aggregated  RAM  values  will  be 
reported  to  the  ASARC/DSARC/IPR  decision 
review  by  the  operational  evaluator  with  the 
rationale  supporting  aggregation.  When  data 
cannot  be  aggregated,  then  both  the  DT  and  the 
OT  results  will  be  presented. 

/.  The  development  tester  or  the  operational 
tester  will  provide  a representative  to  all  scoring 
conferences.  This  representative  will  act  in  an 
advisory  capacity  to  provide  explanations  and 
background  information  on  the  conditions  of 
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tests  for  euch  incident  along  with  resulting 
maintenance  actions  and  hardware  conditions 
or  failure  analysis. 

(j.  The  development  and  operational  testers 
will  determine  initial  failure  chargeability  classi- 
fications. Final  chargeability  classification  is  the 
responsibility  of  the  scoring  conference. 

2-20.  Test  assessment,  a.  The  DT  and  OT  test 
data  base  which  has  been  formally  processed 
through  the  scoring  conference  process  and  the 
aggregated  DT/OT  data  from  the  final  conference 
will  be  used  when  assessing  the,  achievement  of 
RAM  requirements. 

b.  A quantitative  RAM  assessment  of  test 
results  will  be  performed  using  valid  and  appro- 
priate analysis  techniques.  Quantitative  impact 
on  the  analysis  due  to  changes  in  the  assumptions 
(i.e.,  other  distributions)  will  be  assessed.  This 
quantitative  assessment  will  be  included  in  DT 
and  OT  evaluations. 

c.  The  results  of  RAM  testing  will  be  assessed 
and  portrayed  in  test  reports,  independent  evalu- 
ations and  assessments  in  a suitable  form  for  review 
by  and  use  in  the  decision  making  process  (IPR/ 
ASARC/DSARC).  This  wall  include  the  following: 

(1)  Comparison  of  the  demonstrated  relia- 
bility point  estimate  to  the  MAY.  Comparison 
of  the  availability  and  maintainability  point  esti- 
mates with  the  stated  requirements. 

(2)  Portrayal  of  a range  of  reliability  confi- 
dence limits  (one  and  two  sided)  based  on  an  appro- 
priate analysis  of  the  test  data. 

(3)  Display  of  the  user’s  risk  and  degree  of 
confidence  associated  with  the  MAY. 

(4)  A failure  analysis  which  will  identify  the 
frequency  of  part  and  or  failure  mode  occurrences, 
the  impact  of  these  failures  on  mission  performance 
and  logistics  burden,  and  any  safety  considerations 
as  a result  of  critical  and/or  catastrophic  failures. 

(5)  A discussion  of  training  requirements, 
skill  levels,  and  logistic  support  which  are  im- 
pacted because  of  the  RAM  test  results  and  other 
nonmateriel  aspects  of  RAM. 

d.  The  independent  evaluator  for  OT  will  pro- 
vide an  operational  RAM  assessment  of  the  total 
system,  including  all  mission  essential  equipment. 
This  operational  assessment  will  normally  be 
mission  effect  oriented,  will  consider  failures  of  the 
total  system,  and  will  include  incidents  resulting 
from  operator  or  maintenance  errors. 

e.  Based  on  the  DT  and  OT  results,  the  materiel 
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developer  will  provide  an  assessment  as  to  the 
ease  or  difficulty  of  developing  and  incorporating 
design  changes  to  eliminate  selected  failure  modes 
and  the  quantitative  impact,  of  these  changes  on 
reliability. 

/.  The  maintenance  burden  and  the  maintenance 
concept  will  be  assessed  using  developmental  and 
operational  testing  results. 

g.  The  DT  maintainability/maintenance  assess- 
ment will  determine  the  appropriate  maintain- 
ability indices  and  the  degree  of  adherence  to  good 
maintainability  and  human  factors  design  princi- 
ples. DT  maintenance  evaluation  will  also  deter- 
mine whether  the  equipment  publications,  tools, 
test,  measurement  and  diagnostic  equipment  have 
been  developed  to  the  point  that  the  complete 
system  is  ready  for  operational  testing.  The  DT 
maintenance  evaluation  will  be  performed  in  part 
by  military  personnel  under  controlled  conditions. 
No  attempt  will  be  made  to  provide  the  normal 
range  of  personnel  training  or  operational  environ- 
ments during  DT. 

h.  The  OT  maintenance  assessment  will  consider 
the  ability  of  the  using  troops  to  maintain  the 
system  with  the  tools,  equipment  publications  and 
skills  available  in  an  operational  environment.  OT 
will  consider  the  impact  of  the  design  on  the  user, 
and  will  include  a comprehensive  assessment  of 


publications,  tools,  skill  levels  and  allocation  of 
tasks.  Typical  spectra  of  troop  skills  and  varied 
environmental  backgrounds  will  be  normal  com- 
ponents of  OT  maintenance  evaluation. 

2 21.  RAM  assessments,  a.  Throughout  all  phases 
of  the  materiel  life  cycle,  the  materiel  developer 
will  conduct  repeated  RAM  assessments  of  the 
design  and  of  the  hardware’s  measured  and  predi- 
cated RAM  characteristics. 

6.  During  the  course  of  establishing  and  docu- 
menting quantified  RAM  parameters  for  require- 
ments documents,  the  materiel  developer  will 
perform  a baseline  analysis  to  assess  the  technical 
realism  of  proposed  RAM  requirements.  The  pur- 
pose of  a baseline  analysis  is  to  utilize  past  data, 
engineering  analysis,  and  predictions  to  assess 
and/or  challenge  proposed  RAM  parameters.  The 
baseline  analysis  will  also  evaluate  the  suitability 
of  the  proposed  RAM  values,  considering  their 
cost  effectiveness  in  performing  realistically  stated 
missions  or  operating  scenerios.  Such  an  analysis 
on  the  part  of  the  materiel  developer  will  assist  in 
determining  what  quantitative  levels  of  RAM  are 
feasible  and  attainable  within  program  constraints, 
and  determining  what  trade-offs  to  consider  when 
evaluating  the  benefits,  effects,  or  burdens  of 
various  RAM  levels  in  performing  intended  mis- 
sions or  scenerios. 


Section  VI.  NONDEVELOPMENT  ITEMS 


2-22.  General.  Nondevelopmental  items,  include 
the  categories  of  commercial  nondevelopment  items 
(CXDI)  and  military  adaptation  of  commercial 
items  (MACI)  and  items  developed  and  accepted 
by  another  Military  Service  or  government 
agency  or  a foreign  country.  RAM  characteristics 
for  nondevelopmental  items  will  be  consistent 
with  the  standards  of  commercially  designed 
equipment.  Characteristics  required  to  meet  the 
operational  or  logistic  need  will  be  set  forth  in 
terms  currently  used  by  industry  for  that  type  of 
materiel  and  will  be  considered  in  the  technical 
evaluation  associated  with  selection  of  candidate 
items.  Assessments  and  plans  should  indicate  any 
evaluations  of  nondevelopmental  items  and  the 
likelihood  that  these  could  meet  the  operational 
need  including  any  stated  RAM  requirements.  If 
appropriate,  the  nature  of  any  military  adapta- 
tion needed  to  meet  stated  requirements  will  be 
indicated  in  the  technical  assessment  together 
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with  an  evaluation  of  the  effects  of  the  adaptation 
upon  the  RAM  characteristics. 

2-23.  Requirements,  a.  RAM  characteristics  or 
other  suitable  measures  of  effectiveness  essential 
to  meeting  the  operational  and  logistic  needs  must 
be  stated  in  requirements  documents  for  such 
materiel.  Technical  assessments  should  indicate 
pertinent  information  regarding  nondevelopmental 
items  to  include  the  likelihood  that  these  could 
meet  the  required  RAM  and  other  materiel  charac- 
teristics. Technical  assessments  should  also  indi- 
cate the  reasons  for  omission  if  certain  RAM 
characteristics  are  not  included  in  the  require- 
ments document. 

b.  The  materiel  developer  should  give  appro- 
priate consideration  for  using  RAM  warranties  in 
solicitations  and  contracts  for  nondevelopmental 
items. 
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to  type  classification  (Alt  71-6)  to  provide 
satisfactory  evidence  of  acceptability  for  the 
anticipated  Army  utilization  (including  the  opera- 
tional, manpower  and  logistic  environment).  The 
evaluation  will  take  advantage  of  data  from  the 
developing/procuring  agency,  contractors  and/or 
users  of  the  item  to  the  maximum  possible  extent 
in  order  to  avoid  unnecessary  testing.  A Co- 
ordinated Test  Program  (CTP)  will  be  prepared 
in  all  cases  to  identify  required  testing  or  the 
equivalent  evaluation  information  (AR  76-10). 
Results  of  testing  or  equivalent  evaluation  in- 
formation will  be  a consideration  in  the  determina- 


tion of  suitability  made  by  the  appropriate 
decision  review. 

b.  Procurement  documents  will  include  a 
description  of  any  tests  to  be  conducted  and  test 
objectives  and  criteria  to  be  used  to  permit 
certification  that  the  item  satisfies  the  approved 
requirement  and  will  designate  the  organization 
responsible  for  conducting  the  test.  RAM  character- 
istics included  in  documents  used  to  procure 
commerical  nondevelopinental  item-  will  be  stated 
in  terms  consistent  with  those  used  by  industry 
for  that  type  of  materiel.  Similar  consistency 
applies  to  procurement  of  other  nondevelopinental 
items. 


Section  MI.  RECONDITIONED  MATERIEL 


2 25.  General,  a.  The  objective  in  reconditioning 
materiel  is  to  provide  the  user  with  materiel  that 
satisfies  the  user’s  need.  It  is  incumbent  upon 
materiel  developers  to  establish,  in  conjunction 
with  the  user,  appropriate  quantitative  RAM 
requirements  in  the  reconditioned  materiel  tech- 
nical data  or  standard. 

h.  For  new  reconditioned  standards  developed 
on  or  after  tlx1  effective  date  of  this  regulation, 
the  materiel  developer  will  insure  that  RAM  is 
adequately  addressed  in  each  standard.  For  exist- 
ing reconditioned  standards,  the  materiel  de- 
veloper will  establish  a program  to  formally 
review  and  change,  if  necessary,  the  RAM 
requirements  of  reconditioned  standards. 

2-26.  Testing.  The  materiel  developer  will  conduct 


RAM  demonstration  tests  to  insure  that  the 
reconditioned  materiel  meets  the  RAM  require- 
ments stated  in  the  standard. 

2-27.  Assessing,  a.  A product  assessment  will  be 
performed  to  assess  the  total  reconditioning 
process  including  such  things  ns  technical  data, 
kits  and  components,  process  and  procedures, 
configuration  control,  inspection  and  test  equip- 
ment, and  hardware  testing. 

b.  Appropriate  field  data  collection  programs 
will  be  established  to  measure  and  assess  the 
field  performance  of  reconditioned  equipment. 

c.  In  conjunction  with  the  overall  assessment 
of  fielded  items,  a comparison  of  fieid  performance 
for  reconditioned  items  versus  similar  new  pro- 
duction items  will  be  made. 
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A-l.  Annual  support  cost  (ASC).  This  is  the  direct, 
annual  cost  of  maintenance  personnel,  repair,  parts 
and  transportation  for  all  corrective  (either  on- 
system,  off-system,  or  both)  and  preventive 
maintenance  actions  when  the  system  operates 
“X”  hours  per  years  during  the  “Nth”  year  of 
“M”  years  service  life  where  the  system  is  defined 
as  Y units  of  item  A,  Z units  of  item  B,  etc. 

A-2.  Assess.  Make  an  objective  determination  of 
the  degree  to  which  the  system  under  test  met 
specified  performance  criterion,  based  on  the  facts 
available  from  testing. 

A-3.  Availability.  A measure  of  the  degree  to 
which  an  item  is  in  operable  and  committable  state 
at  the  start  of  the  mission,  when  the  mission  is 
called  for  at  an  unknown  (random)  point  in  time 
(MIL-STD-721B). 

A-4.  Availability  (Inherent).  The  percentage  of 
time  the  system  is  operating  when  considering 
only  operational  time  and  unscheduled  (corrective) 
maintenance  time.  The  formula  is: 

OT 

' OT+TCM 

Where : 

OT  = The  operating  time  during  a given  calendar 
time  period. 

TCM=Tho  total  corrective  maintenance  time  in 
clock  hours  during  the  stated  period. 

A-5.  Availability  (Achieved).  The  percentage  of 
time  the  system  is  operating  when  considering 
only  operating  time  and  total  maintenance 
(scheduled  and  unscheduled)  time.  The  formula 
is: 

OT 

A*  OT+TCM  + TPM 

Where : 

OT,  TCM  are  defined  in  A-4  above. 

TPM  = The  total  preventive  maintenance  time  in 
clock  hours  during  the  stated  OT  period. 

A-6.  Availability  (Operational).  This  parameter  is 
a measure  of  the  degree  to  which  an  item  is 
either  operating  or  is  capable  of  operating  at  any 
random  point  in  time  when  used  in  a typical 
maintenance  and  supply  environment.  This  defi- 
nition covers  total  calendar  time  where: 
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OT  -r  ST 

A"  “ OT  + ST  + TCM  -t-  TPM + TA  DLT 

= Total  calendar  time— total  downtime 
Total  calendar  time 

OT= Operating  time  per  given  calendar  time  period. 

ST  = Standby  time  (not  operating,  but  assumed 
operable)  per  given  calendar  time  period. 

TCM= Total  corrective  (unscheduled)  maintenance 
time  per  given  calendar  time  period. 

TPM  = Total  preventive  (scheduled)  maintenance  time 
per  given  calendar  period. 

TALDT  = Total  administrative  and  logistics  downtime 
spent  waiting  for  parts,  maintenance  personnel 
or  transportation  per  given  calendar  time  period. 

A-7.  Combat  developer.  The  command  or  agency 
responsible  for  the  formulation  of  concepts 
doctrine,  organization  (excluding  Army  wholesale 
logistics)  and  materiel  objectives,  and  require- 
ments for  the  employment  and  support  of  US 
Army  forces  in  a theater  of  operations  and  in  the 
control  of  civil  disturbances.  The  combat  developer 
formulates  Army  management  systems  (in  logis- 
tics, personnel,  administrative,  training,  and  other 
functional  areas  as  designated)  which  impact 
directly  on  or  extend  into  a theater  of  operations. 
The  combat  developer  formulates  the  Army 
structure  (US  Army  Training  and  Doctrine 
Command  (TRADOC)  is  the  Army’s  principal 
combat  developer.). 

A-8.  Critical  item.  Any  part  or  item  that  (1) 
prevents  satisfactory  operation  of  the  system  or 
creates  unwarranted  safety  hazards  if  it  fails  to 
function,  (2)  is  sufficiently  complex  to  warrant 
special  production  techniques  or  controls,  (3)  re- 
quires special  treatment  or  handling  during  trans- 
port or  storage,  (4)  imposes  a heavy  maintenance 
and  supply  support  burden,  or  (5)  has  a long 
production  lead  time. 

A-9.  Development.  Systematic  use  of  scientific 
knowledge  directed  toward — 

a.  Improving  or  creating  useful  products  to 
meet  specific  performance  requirements  but  ex- 
cluding manufacturing  and  production  engineer- 
ing. 

b.  Creating  components  for  end  items  to  meet 
specific  performance  requirements. 
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A-10.  Development  plan  (DP).  A document  which 
records  program  decision,  contains  the  user’s 
requirement,  provides  appropriate  analysis  or 
technical  options,  and  the  life  cycle  ])lans  for 
development,  testing,  production,  training,  and 
logistic  support  of  materiel  items  (AR  70-27). 
A-ll.  Durability.  A special  case  of  reliability 
(MIL- -STD-72111).  Durability  is  the  probability 
that  an  item  will  successfully  survive  its  projected 
life,  overhaul  point,  or  rebuild  point  (whichever  is 
the  more  appropriate  durability  measure  for  the 
item)  without  a durability  failure.  A durability 
failure  is  considered  to  be  a malfunction  that 
precludes  further  operation  of  the  item  ami  is 
great  enough  in  cost,  safety,  or  time  to  restore, 
that  the  item  must  be  replaced  or  rebuilt. 

A-12.  Effectiveness.  A measure  of  the  extent  to 
which  an  item  satisfies  a set  of  specific,  pre- 
established  requirements. 

A-13.  Failure.  The  inability  of  an  item  to  perform 
within  previously  specified  limits  (MIL-STD- 
721B). 

A-14.  Failure  modes.  The  ways  in  which  an  item 
can  fail,  including  the  type  of  failure  (e.g.,  short, 
open,  fatigue)  and  degree  of  failure  (e.g.,  partial, 
total). 

A-15.  Item.  A unit  of  materiel  at  any  level  of 
assembly  (e.g.,  weapon  system,  subsystem,  com- 
ponent) (MIL-STD-721B). 

A-16.  Life  cycle.  All  phases  through  which  an 
item  passes  from  conception  through  disposition 
(DA  Pam  11-25). 

A-17.  Maintainability.  A characteristic  of  design 
and  installation  which  provides  inherently  for  the 
item  to  be  retained  in  or  restored  to  a specified 
condition  within  a given  time,  when  the  main- 
tenance is  performed  in  accordance  with  pre- 
scribed procedures  and  resources. 

A-18.  Maintainability  engineering.  The  applica- 
tion of  scientific  knowledge  and  engineering  skills 
to  the  development  of  items  of  Army  equip- 
ment to  provide  an  inherent  ability  to  be  main- 
tained (i.e.,  favorable  maintenance  characteristics). 
A-19.  Maintenance.  All  actions  necessary  for 
retaining  an  item  in  or  restoring  it  to  a specified 
condition  (M [Lr-STD-72lB). 

A 20.  Materiel  developer.  The  agency  responsible 
for  research,  development  and  production  valida- 
tion of  an  item  (to  include  the  system  for  its 
logistical  support)  which  responds  to  DA 
objectives  and  requirements  (AR  70-1). 

A-2 


A-21.  Mean-time-between-failure  (MTBF).  For  a 

particular  interval,  the  total  functioning  life  of  an 
item  divided  by  the  total  number  of  failures 
during  the  measurement  interval.  The  definition 
applies  to  time,  cycles,  distance,  events,  or  other 
measures  of  life  units  (MTL-STD-721B). 

A-22.  Mean-time-to-repair  (MTTR).  The  total 
corrective  maintenance  time  divided  by  the  total 
number  of  corrective  maintenance  actions  during 
a given  period  of  time  (MIL-STD-721B). 

A-23.  Mission  profile.  A description  of  the  opera- 
tional requirement(s)  that  a system  must  com- 
plete to  accomplish  a particular  mission  or  set  of 
missions.  This  profile  will  be  used  as  the  basis 
for  mission  reliability'  assessment.  The  tasks  may 
be  multifunctional  (an  item  performing  several 
tasks  such  as  a tank  shooting,  moving,  and  com- 
municating) ; single-function  continuous  (an  item 
continuously  performing  one  task  such  as  a 
surveillance  radar) ; single-function  cyclic  (an 
item  performing  the  same  task  repeatedly  such 
as  a missile  launcher  or  artillery  piece) ; or  single- 
function one-time  (an  item  performing  only  a 
one-time  task  such  as  a missile  or  munition)  and 
described  in  terms  such  as  hours,  miles,  or  rounds 
fired. 

A-24.  Operational  mode  summary.  A description 
of  the  anticipated  mix  of  ways  the  equipment  will 
be  used  in  carrying  out  its  operational  role. 
Includes  expected  percentage  of  use  in  each  role 
and  percentage  of  time  it  will  be  exposed  to  each 
type  of  environmental  condition  during  the 
system  life. 

A-25.  Operational  performance  characteristics. 

System  characteristics  fundamental  to  the  effec- 
tiveness of  a mission,  including  characteristics 
such  as  speed,  range,  and  accuracy. 

A-26.  Reliability.  The  probability  that  an  item 
will  perform  to  its  intended  function  for  a specified 
interval  under  stated  conditions  (MIL-STD- 
721B). 

A-27.  Specified  value.  The  value  specified  in  the 
contract  or  equipment  specification  (MIL-STD- 
781B). 

A-  28.  Supporting  agency.  The  agency  responsible 
for  planning  or  carrying  out  integrated  logistic 
support  embracing  one  or  more  of  the  following 
disciplines : Maintenance,  supply  (including  pro- 
curement), transportation,  personnel,  training, 
facilities,  and  data  collection  and  analysis. 
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A-29.  Systems  engineering.  The  application  of 
scientific  and  engineering  knowledge  to  the 
planning,  design,  construction,  and  evaluation  of 
man-machine  systems  and  components.  It  includes 
the  overall  consideration  of  possible  methods 
for  accomplishing  a desired  result,  determination 
of  technical  specificaton,  identification  and  solu- 
tion of  problems  between  parts  of  the  system, 
development  of  coordinated  test  programs,  assess- 


ment of  data,  integrated  logistic  support  planning, 
and  supervision  of  design  work. 

A-  30.  Technical  performance  characteristics.  Sys- 
tem characteristics  that  describe  aspects  of  the 
system  such  as  reliability,  maintainability,  safety 
factor,  and  electromagnetic  field. 

A-31.  Thresholds.  A minimum  or  maximum  value 
established  for  each  characteristic  which,  when 
breached,  will  cause  a program  review  (ASAItC/ 
DSAKC/IPJi,  ns  appropriate). 
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DA  Pam  11-25 


Department  of  the  Army  Productivity  Improvement 
Program 

Organization  and  Functions  Department  of  the  Army 
Army  Electromagnetic  Compatibility  Program 
Logistic  Readiness 

Systems  Acquisition  Review  Council  Procedures 
Army  Research,  Development,  and  Acquisition 
Grants  to  Nonprofit  Organizations  for  Support  of  Scientific 
Research 

Human  Resources  Research  Program 

Test  and  Evaluation  During  Development  and  Acquisition 
of  Materiel 

Product  Improvement  of  Materiel 

Outline  Development  Plan/Development  Plan/Army  Pro- 
gram Memorandum/Defense  Program  Memorandum,' 
Decision  Coordinating  Paper 
Configuration  Management 

Research,  Development,  Test,  and  Evaluation  of  Materiel 
for  Extreme  Climatic  Conditions 
Army  Combat  Developments 
User  Testing 

Introduction  of  New  or  Modified  Systems/Equipment 
Type  Classification/Reclassification  of  Army  Materiel 
Military  Training  Aids  and  Army  Training  Aids  Centers 
Materiel  Objectives  and  Requirements 
Army  Aircraft  Inventory,  Status,  and  Flying  Time,  Re- 
quirement Control  Symbol  (RCS)  DRC-130 
System  Safety 

Human  Factors  Engineering  Program 
Defense  Standardization  Program 
Army  Data  Management  Program 
Army  Industrial  Preparedness  Program 
Integrated  Logistic  Support 
Proposal  Evaluation  and  Source  Selection 
Army  Materiel  Maintenance  Concepts  and  Policies 
Sample  Data  Collection — The  Army  Maintenance  Manage- 
ment System  (TAMMS) 

Test,  Measurement,  and  Diagnostic  Equipment 
Basic  Policies  for  Systems  Acquisition  by  the  Department 
of  the  Army 

Life  Cycle  System  Management  Model  for  Army  Systems 
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The  proponent  agency  of  this  regulation  is  the  US  Army  Materiel  Develop- 
ment and  Readiness  Command.  Users  are  invited  to  send  comments  and 
suggested  improvements  on  DA  Form  2028  (Recommended  Changes  to 
Publications  and  Blank  Forms)  to  Cdr,  DARCOM,  ATTN:  DRCQA— E,  5001 
Eisenhower  Ave,  Alexandria,  VA  22333. 
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RELIABILITY  DEVELOPMENT  TESTS 


MIL -STD-2068 (AS) 

1.  This  Military  Standard  has  been  approved  for  use  by  the  Naval  Air 
Systems  Command,  Department  of  the  Navy,  and  is  available  for  use 
by  all  Departments  and  Agencies  of  the  Department  of  Defense. 

2.  Beneficial  comments  (recommendations,  additions,  deletions)  and 
any  pertinent  data  which  may  be  of  use  in  improving  this  docu- 
ment should  be  addressed  to  Engineering  Specifications  and  Stan- 
dards Department  (Code  93)  Naval  Air  Engineering  Center,  Lake- 
hurst,  NJ  08733. 
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FOREWORD 


The  reliability  achieved  by  military  systems  is  directly  dependent  upon 
the  reliability  requirements  imposed,  and  upon  the  emphasis  placed  on 
reliability  by  (government  and  contractor)  management  throughout  the 
life  cycle.  MIL-STD-785  provides  criteria  for  a reliability  program  to 
aid  in  the  timely  and  economical  attainment  of  reliability  requirements. 
The  purpose  of  this  standard  is  to  amplify  the  requirements  for  the 
reliability  development  testing  included  in  the  reliability  program 
covered  by  MIL-STD-785.  The  test  program  resulting  from  implementation 
of  this  standard  should  be  coordinated  and  integrated  with  other  design, 
development,  and  production  functions  to  permit  the  most  economical 
achievement  of  overall  program  objectives. 
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1.  SCOPE 

1.1  Purpose.  This  standard  established  requirements  and  procedures 
for  i reliability  development  test  to  implement  the  MIL-STD-735  re- 
quirement for  such  a test.  The  purpose  of  the  reliability  development 

test  is  reliability  growth  and  assessment  to  promote  reliability  improvement 
of  systems  and  equipments  in  an  orderly  and  standardized  manner. 

1.2  Application.  This  standard  is  applicable  to  Naval  Air  Systems 
Command  procurements  for  development  of  systems  and  equipment.  This 
standard  addresses  only  the  reliability  development  test  phase  of 
MIL-STD-785  and  the  reliability  program  plan.  The  reliability  develop- 
•ment  tests  do  not  replace  the  design,  performance,  environmental,  pre- 
production,  individual,  qualification,  or  other  required  tests  specified 
for  the  system  or  equipment.  In  accordance  with  the  approved  reliability 
program  plan,  tasks  such  as  those  described  in  Appendix  A shall  be 
completed  and  approval  obtained  from  the  procuring  activity  prior  to  the 
initiation  of  reliability  development  tests.  This  standard  does  not 
apply  to  "single-shot"  and  passive  use"  devices  such  as  cartridges, 
helmets,  garments,  etc. 


2.  REFERENCED  DOCUMENTS 


2.1  Issues  of  documents.  The  following  documents  of  the  issue  in 
effect  on  the  date  of  invitation  for  bids  or  request  for  proposal,  form 
a part  of  this  standard  to  the  extent  specified  herein. 


SPECIFICATIONS 

MILITARY 

MIL-C-45662  Calibration  System  Requirement 
STANDARDS 
MILITARY 


MIL-STD-230 

MIL-STD-A80 

MI L -STD- 721 

MIL-STD-781 
MIL-STD-785 
MIL-STD-81 0 


Definitions  of  Item  Levels,  Item  Exchangeability, 
Models  and  Related  Terms 

Configuration  Control  - Engineering  Chages,  Deviations 
and  Waivers 

Definitions  of  Effectiveness  Terms  for  Reliability, 
Maintainability,  Human  Factors  and  Safety 
Rel iability  Tests:  Exponential  Distribution 

Reliability  Program  for  Systems  and  Equipment 
Environmental  Test  Methods 


HANDBOOKS 


MILITARY 


MIL-HDBK-472  Maintainability  Prediction 
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PUBLICATIONS 

NAVAL  AIR  SYSTEMS  COMMAND 

AR21  Aeronautical  Requirements  Ground  Support  Equipment 

(Copies  of  specif ications , standards,  drawings,  and  publications  required 
by  contractors  in  connection  with  specific  procurement  functions  should 
be  obtained  from  the  procuring  activity  or  as  directed  by  the  contracting 
officer. ) 

DEFINITIONS 

3.1  Terms.  Meanings  of  terms  not  defined  herein  are  in  accordance 
with  the  definitions  in  MIL-STD-721,  MIL-HDBK-472,  and  AR-21 . 

3.1.1  Contractor.  The  term  "contractor"  is  defined  as  any  corporation, 
company,  association,  or  individual  which  undertakes  performance  under 
the  terms  of  a contract,  letter  of  intent  or  purchase  orders,  project 
orders,  and  allotment,  in  which  this  document  may  be  incorporated  by 
reference.  For  the  purpose  of  this  document,  the  term  "contractor" 
includes  government  operated  activities  under  an  airtask,  project  order, 
or  allotment. 

3.1.2  Corrective  action.  A documented  design,  process,  procedure,  or 
materials  change  to  correct  the  true  cause  of  a failure.  Part  replace- 
ment with  a like  item  does  not  constitute  appropriate  corrective  action. 

3.1.3  Failure.  Details  involving  failure  criteria  must  be  stated  in 
the  equipment  specification  and  test  procedures.  For  test  purposes 
failure  shall  include: 

a.  The  inability  of  an  item  to  perform  its  required  function 
within  specified  limits  under  stated  conditions.  Failure  includes 
intermittent  cessation  of  performance  or  lack  of  response  to  a command. 

b.  Mechanical  or  structural  parts  or  components  of  an  item  broken, 
fractured,  or  damaged  during  environmental  testing  or  operation  within 
specified  stress  limits. 

3.1.4  Environments.  The  conditions,  circumstances,  influences, 
stresses  and  combinations  thereof,  surrounding  and  affecting  systems  or 
equipment  during  storage,  handling,  transportation,  testing,  installa- 
tion, and  during  use  in  standby  status  and  mission  operation. 

3.1.5  Failure  analysis.  The  identification  of  the  failure  mode,  the 
failure  mechanism,  and  the  cause  (i.e.,  defective  soldering,  design 
weakness,  contamination,  assembly  techniques,  etc.). 
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3.1.6  Mechanism  of  failure.  The  original  defect  which  initiated  the 
failure  or  the  physical  process  by  which  the  degradation  proceeded  to 
the  point  of  failure,  identifying  quality  defects,  internal,  structural, 
or  electrical  weakness  and,  where  applicable,  the  nature  of  externally 
applied  stresses  which  led  to  failure. 

3.1.7  Mode  of  failure.  The  effect  by  which  a failure  is  observed. 
Generally,  describes  the  way  the  failure  occurs  and  tells  "how"  with 
respect  to  operation. 

3.1.8  Predicted  reliability.  That  reliability  which  is  expected  at 
some  future  date,  postulated  on  analysis  of  the  design  and  the  failure 
rates . 

3.1.9  Reliability  assessment.  An  estimate  of  the  achieved  reliability 
obtained  by  calculation  using  data  gathered  during  tests. 

3.1.10  Reliability  growth.  The  positive  improvement  of  the  reliability 
of  equipment  through  the  systematic  and  permanent  removal  of  failure 
mechanisms,  regardless  of  their  sources,  by  taking  corrective  action. 

3.1.11  Specified  mean  time  between  failure  (Q) . The  MTBF  value 
specified  in  the  equipment  specification  or  contract. 
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4.  GENERAL  REQUIREMENTS 


r 


4.1  General . The  reliability  development  test  shall  be  planned  and 
conducted  in  accordance  with  this  standard  to  provide  engineering  in- 
formation on  failure  modes  and  mechanisms  under  natural  and  induced 
environments,  to  provide  information  needed  for  assessing  the  reliability 
of  equipment,  and  to  provide  for  reliability  improvement  by  the  iterative 
process  of  "test,  analyze,  and  fix".  The  program  required  by  this 
standard  differs  from  that  required  by  MIL-STD-781  in  that  this  program 
does  not  have  as  an  objective  the  demonstration  of  any  specific  quantitative 
reliability  requirement  or  the  acceptance  of  hardware,  but,  rather,  it 

has  as  an  objective  the  assurance  that  the  majority  of  the  reliability 
problems  h„ve  been  resolved. 

4.2  Test  plan.  The  contractor  shall  prepare  a reliability  development 
test  plan  and  obtain  approval  from  the  procuring  or  user  activity  prior 

to  the  commencement  of  testing.  The  test  pain  shall  reflect  the  requirements 
of  the  contract,  equipment  specification  and  this  document.  Test  planning 
shall  consider  the  possibility  of  special  tests  to  verify  corrective 
action  implementation  and  to  obtain  test  data  that  reflect  the  effectiveness 
of  corrective  action  implementation.  The  test  plan  shall,  as  a minimum, 
include  the  following: 

a.  Test  objectives  and  requirements  including  planned  growth, 

b.  Test  item  description  and  number  to  be  tested, 

c.  Test  conditions,  environmental,  operational,  and  performance 
profiles,  and  the  duty  cycle, 

d.  Test  schedules  and  milestones,  including  the  test  program 
review  schedule, 

e.  Test  conduct  ground  rules,  failure  criteria,  interface  boundaries, 
and  "No  Test"  criteria, 

f.  Test  facility  and  equipment  descriptions  and  requirements, 

g.  Data  collection  and  recording  requi rements , 

h.  Analysis  requirements  and  methods  of  calculation, 

i.  Government  Furnished  Property  requirements  and  impact, 

j.  Description  of  preventive  maintenance  to  be  accomplished  during 
test,  and 

k.  Final  restoration  and  disposition  of  test  items. 


4.3  Test  levels.  The  test  levels  describe  the  condition  of  temperature, 
vibration,  humidity,  equipment  inputs  (e.g.,  input  voltage,  hydraulic 
pressure,  torque,  etc.),  and  the  cycling  of  these  conditions  to  be 
applied  to  the  test  items(s)  during  the  test.  The  equipment  inputs,  the 
environmental  conditions,  and  the  combinations  of  environmental  types 
and  levels  to  be  applied  during  tests  and  their  variation  as  a function 
of  test  time,  shall  be  as  specified  in  the  equipment  or  model  specification 
and  described  in  the  test  plan.  The  specified  test  levels  shall  be 
tailored  to  the  system  or  equipment  mission  and  environmental  profiles. 

The  following  conditions  are  representative,  but  not  necessarily  all 
i ncl usi ve. 
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4.3.1  Temperature.  The  temperature  requirements , including  temperature 
cycling  shall  be  as  specified  in  the  equipment  or  model  specification 

and  described  in  the  test  plan.  Unless  otherwise  specified,  a thermal 
survey  shall  be  made  in  accordance  with  MI L -STD- 781  prior  to  the  start 
of  the  reliability  development  tests  using  the  hioh  and  low  storage 
temperature  as  specified  in  the  equipment  or  model  specification.  (See 
5.1.3) 

4.3.2  Vibration.  Unless  otherwise  specified,  the  test  item  shall  be 
subjected  to  externally  imposed  random,  psendo-random,  complex  motion, 
or  acoustic  vibration  excitation.  The  levels,  tolerances  ar.d  durations 
shall  be  as  specified  in  the  equipment  specification.  The  mounting  of 
the  test  item  shall  be  in  accordance  with  MIL-STD-810,  Method  514. 

Unless  otherwise  specified,  a vibration  survey  to  determine  test  item 
resonances  shall  be  made  in  accordance  with  the  requirements  of  MIL-STD- 
810  prior  to  the  start  of  the  reliability  development  tests. 

4.3.3  Equipment  inputs.  Unless  otherwise  specified,  the  inputs 
(voltage  hydraulic  pressure,  torque,  etc.)  to  the  test  item  shall  be 
varied  above  and  below  the  nominal  values  to  the  levels  specified  in  the 
equipment  or  model  specification.  Each  input  value  (maximum,  nominal, 
and  minimum)  shall  be  maintained  for  one-third  of  the  equipment  on  time. 

This  cycling  procedure  is  to  be  repeated  continuously  throughout  the 
test  as  described  in  the  approved  test  plan. 

4.3.3. 1 On-Off  cycl ing.  For  test  purposes,  the  number  of  times  that 
equipment  will  be  turned  ON  for  preoperational  checks  or  run-ups,  plus 
the  number  of  times  turned  ON  and  OFF  from  the  completion  of  one  opera- 
tional mission  to  the  completion  of  the  next,  shall  be  as  specified  in 
the  equipment  or  model  specification.  Additional  On-Off  cycling  for 
normal  maintenance  and  repair  shall  not  be  included. 

4. 3. 3. 2 Duty  cycle  . The  equipment  duty  cycle,  the  time  phase 
apportionment  of  modes  of  operation  and  function(s)  to  be  performed  by 
the  test  item  during  test,  shall  be  as  specified  in  the  equipment  or 
model  specification.  Alternative  duty  cycles,  when  applicable,  shall 
also  be  as  specified. 

4.3.4  Humid i ty . Humidity  shall  be  as  specified  for  condensation, 
frosting,  and  freezing  conditions  expected  during  service  use  of  the 
equipment  under  test.  Humidity  levels  need  not  be  controlled  during  the 
test  but,  at  appropriate  times  during  the  test,  moisture  may  be  injected 
to  obtain  the  specified  condensation,  frosting,  or  freezing  conditions. 

4.4  Test  facilities  and  apparatus.  Test  facilities,  chambers,  and 
apparatus  used  in  conducting  the  tests  shall  be  capable  of  providing  and 
maintaining  the  conditions  specified  for  the  applicable  test  levels.  The 
test  facilities  and  apparatus  shall  be  described  in  the  test  plan. 

4.4.1  Tolerance  of  test  conditions.  Unless  otherwise  specified, 
tolerances  of  test  conditions  shall  be  as  specified  in  MIL-STD-810. 
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4.4.2  Accuracy  of  test  apparatus.  The  accuracy  of  instruments  and 
other  test  equipment  used  to  control  or  monitor  the  test  parameters 
shall  be  verified  periodically  in  accordance  with  the  requirements  of 
MIL-C-45662  and  to  the  satisfaction  of  the  procuring  or  user  activity. 

4.5  Performance  of  test.  The  electrical  and  mechanical  performance 
outputs  and  characteristics  to  be  monitored  and  measured  before,  during, 
and  after  the  tests,  including  the  quantitative  range  of  acceptable  per- 
formance for  each  parameter,  shall  be  as  defined  in  the  equipment  or 
model  specification  and  described  in  the  test  plan.  Any  performance 
outside  these  limits  shall  be  recorded  as  a problem  or  failure,  whether 
or  not  the  equipment  ceases  to  operate. 

4.5.1  Pretest  performance.  Unless  otherwise  specified,  prior  to  con- 
ducting any  test,  the  test  item  shall  be  tested  and  a record  shall  be 
made  of  all  data  to  determine  compliance  with  required  performance  and 
to  provide  reference  levels  or  criteria  for  checking  desired  performance 
of  the  test  item  during  and  at  the  conclusion  of  the  tests.  The  pretest 
performance  check  may  be  made  after  installation  of  the  item  in  the  test 
facility  if  installation  conditions  necessitate  it. 

4.5.2  Installation  of  test  item  in  test  facility.  Unless  otherwise 
specified,  the  test  item  shall  be  installed  in  the  test  facility  in  the 
manner  described  in  MI L-STD-8 1 0 . After  test  item  installation,  the 
equipment  and  the  test  facility  shall  be  operated  to  determine  that  the 
test  set-up  operates  properly  under  the  required  test  conditions. 

4.5.3  Performance  during  test.  The  performance  measurements  taken 
during  the  test  shall  be  as  specified  in  the  equipment  or  model  specifica- 
tion and  described  in  the  test  plan.  The  specified  measurements  taken 
during  the  test  shall  be  compared  to  the  pretest  data  to  determine 
whether  test  exposure  is  producing  changes  in  performance.  The  specified 
performance  parameters  shall  be  monitored  so  as  to  record  problems  or 
failures  at  the  time  they  occur  during  the  duty  cycle.  Periodic  monitoring 
schemes  appropriate  to  operational  use  may  be  used  when  approved  by  the 
procuring  activity.  Built-in  test  (BIT)  included  as  part  of  the  equip- 
ment design  shall  be  used  as  part  of  the  problem  or  failure  detection 
method . 

4.5.4  Post-test  performance.  Unless  otherwise  specified,  the  test 
item  shall  be  tested  at  the  completion  of  the  test,  and  a record  shall 
be  made  of  all  data,  and  the  results  compared  to  the  pretest  data  to 
determine  any  change  in  performance. 

4.5.5  Maintenance.  Maintenance  during  this  test  shall  be  limited  to 
repair  of  failures,  correction  of  failures,  and  the  preventive  main- 
tenance specifically  defined  in  the  test  plan.  If  parts  are  replaced 
based  on  the  preventive  maintenance  defined  in  the  test  plan,  such 
replacement  shall  be  classified  as  a failure  and  appropriate  action 
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shall  be  taken,  but  such  failure  shall  not  be  counted  for  purposes  of 
the  reliability  assessment.  Adjustments  of  controls  which  could  not  be 
accomplished  readily  during  a mission  shall  be  classified  as  a failure 
and  appropriate  action  shall  be  taken.  Unless  otherwise  specified,  the 
maintenance  time  for  each  problem,  failure,  or  maintenance  action  shall 
be  documented  for  inclusion  in  the  final  report  (see  4.9.3).  The  maintenance 
time  shall  be  subdivided  into  the  following  categories  as  defined  in 
MIL-HDBK-472: 

a.  Localization, 

b.  Isolation, 

c.  Disassembly, 

d.  Interchange, 

e.  Reassembly, 

f.  Alighment, 

g.  Check-out,  and 

h.  Total 

4.5.6  Test  records.  Test  records  shall  be  maintained  continuously  and 
as  specified  in  the  equipment  or  model  specification  and  described  in 
the  test  plan.  When  not  specified,  they  shall  be  maintained  in  oeneral 
accord  with  the  following. 

4. 5. 6.1  Test  log  and  data  record.  The  test  log  and  data  record  shall 
be  a complete  record  of  required  test  data  for  each  item  under  test  and 
shall  be  maintained  throughout  the  test.  The  format  shall  permit  ready 
reference  of  the  test  history  of  each  test  item.  Completed  test  log 
sheets  may  be  duplicated  but  not  recopied  prior  to  submission  to  the 
procuring  activity.  An  entry  shall  be  made  in  the  test  log  and  data 
record  each  time  a test  item  is  checked.  The  date,  time,  time  meter 
reading,  the  environmental  conditions,  and  the  exact  values  of  any 
parameters  measured  shall  be  recorded. 

4. 5. 6. 2 Test  item  failure  record.  A failure  record  shall  be  main- 
tained for  each  test  item.  The  record  shall  be  designed  to  permit 
keeping  the  entire  test  history  of  each  test  item  on  a single  sheet  so 
that  widely  divergent  differences  in  test  behavior  between  test  items 
may  be  easily  recognized.  The  test  item  problem  and  failure  record 
shall  be  prepared  as  described  in  the  test  plan  and  an  entry  made  on  the 
occurrence  of  each  problem  or  failure. 

4. 5. 6. 3 Failure  tag.  A failure  tag  shall  be  affixed  to  the  test  item 
immediately  upon  detection  of  any  problem  or  failure.  The  failure  tags 
will  provide  space  for  the  problem  and  failure  report  serial  number  and 
for  other  pertinent  entries  from  the  test  log  and  test  item  failure 
record.  Failure  tags  shall  be  prepared  as  described  in  the  test  plan. 

4.6  Failure  reporting,  analysis,  and  corrective  action.  The  con- 
tractor shall  implement  and  maintain  and  shall  require  subcontractors 
also  to  implement  and  maintain  a comprehensive  system  for  identifying. 


MIL -STD- 2068 (AS) 

21  March  1977 

reporting,  investigating,  analyzing,  and  correcting  all  problems  and 
failures  which  occur  throughout  the  reliability  development  test  program. 

The  contractor's  existing  failure  reporting,  analysis,  and  corrective 
action  system  may  be  utilized  with  minimum  changes  necessary  to  meet  the 
requirements  of  this  document.  The  system  shall  cover  all  test  items 
and  the  interfaces  between  the  test  items  and  the  test  equipment,  test 
procedures,  test  personnel  and  the  handling  and  operating  instructions. 

The  system  shall  provide  for  investigation  and  engineering  analysis  of 
each  problem  or  failure,  followed  where  appropriate  by  laboratory  analysis 
of  failed  items. 

4.6.1  Problem  and  failure  action.  At  the  occurrence  of  a problem 
that  affects  satisfactory  operation  of  the  equipment  as  described  in  the 
test  procedure,  or  a failure,  entries  shall  be  made  on  the  appropriate 
data  logs  and  the  equipment  removed  from  test.  After  this  equipment  has 
been  returned  to  specified  operating  condition,  it  may  be  returned  to 

test  with  appropriate  entries  in  the  data  logs.  Unless  otherwise  specified, 
temporary  replacement  of  plug-in  items  is  authorized  to  permit  test 
continuation  while  troubleshooting,  repair,  and  analysis  are  performed. 
Individual  acceptance  tests  on  repaired  items  shall  be  performed  prior 
to  reinstallation  in  the  test  facility. 

4.6.2  Problem  and  failure  report.  A problem  and  failure  report  shall 
be  prepared  at  the  occurrence  of  each  problem  or  failure.  The  report 
shall  contain  the  necessary  information  to  permit  determination  of  the 
origin  and  possible  correction  of  failures.  The  contractor's  existing 
problem  and  failure  report  form  may  be  used  with  minimum  changes  necessary 
to  meet  the  requirements  of  this  document.  The  information  to  be  recorded 
shall  include,  as  a minimum,  the  following: 

a.  Information  wlv'ch  will  indicate  symptoms  of  failure,  test 
conditions,  identification  of  failed  equipment,  total  test  time,  and 
test  item  operating  time  at  time  of  failure  shall  be  recorded  at  the 
time  the  problem  and  failure  report  is  initiated. 

b.  Information  for  each  independent  and  dependent  failure 

and  the  extent  of  confirmation  of  the  failure  symptoms,  the  identification 
of  the  failures,  and  a description  of  all  repair  action  taken  shall  be 
recorded  at  the  time  of  repair. 

c.  Information  which  indicate  the  results  of  the  investiga- 
tion, the  analysis  of  all  part  failures,  an  analysis  of  the  equipment 
design,  and  the  corrective  engineering  action  taken  shall  be  recorded  at 
the  completion  of  the  failure  investigation.  If  no  corrective  engineering 
action  is  taken,  then  such  decision  must  be  justified. 

4.6.3  Investigation  and  analysis  of  problems  and  failures.  The  cause 
of  each  problem  and  failure  shall  be  determined  by  investigation  and 
analysis.  The  investigation  and  analysis  shall  be  adequate  to  assess 
causes,  mechanisms,  and  potential  effects  of  the  problem  or  failure  and 
to  serve  as  a basis  for  decision  on  the  corrective  actions.  Investiqa 
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tion  and  analysis  shall  consist  of  any  applicable  method  (such  as  test, 
application  study,  dissection,  x-ray  analysis,  microscopic  analysis, 
spectrographic  analysis,  etc.)  which  may  be  necessary  to  determine  the 
cause  of  failure.  The  investigations  and  their  conclusions  shall  be 
documented  and  shall,  as  a minimum,  provide  the  following  information: 

a.  Failure  date  and  time, 

b.  Necessary  design  changes  have  been  devised  and  accomplished, 

c. .  Necessary  design  changes  have  been  verified  by  test, 

d.  Effectivity  of  corrective  action  has  been  established, 

e.  The  corrective  action  has  been  made  in  existing  identical 
items  to  which  it  is  pertinent, 

f.  The  closeout  document  has  been  signed  off  by  the  appropriate 
contractor  technical  management  authority  to  indicate  technical 
review  and  to  certify  completion  of  all  closeout  actions,  and 

g.  Procuring  or  user  activity  has  concurred  in  all  actions  taken 
or  to  be  taken. 

4.6.4  Corrective  action.  When  the  cause  of  a problem  or  failure  has 
been  determined,  a corrective  action  shall  be  developed  to  reduce  or 
prevent  the  recurrence  of  the  problem  or  failure.  The  effectiveness  of 
the  corrective  action  shall  be  demonstrated  by  being  tested  under  the 
same  controlled  conditions  under  which  the  original  failure  was  observed. 
Unless  otherwise  specified,  change  control  procedures  shall  be  in  accordance 
with  MIl-STD-480  change  procedures  for  the  engineering  and  operational 
testing  phases  of  a program. 

4.6.5  Classification  of  failures.  For  the  purpose  of  reliability 
growth  assessment,  all  failures  shall  be  classified  as  specified  in 
Appendix  B. 

4.7  Reliability  growth  assessment.  Reliability  growth  assessment  is 
obtained  by  calculation  using  data  obtained  from  tests.  The  reliability 
of  the  equipment  shall  be  monitored  throughout  the  test  by  graphic 
plots.  The  vertical  axis  of  the  graph  will  show  the  appropriate  reliability 
parameter  (MTBF,  probability  of  mission  success,  etc.)  of  the  item.  The 
horizontal  axis  will  be  in  units  of  cumulative  test  time  (hours,  cycles, 
rounds,  multiples  of  MTBF,  etc.)  The  specified  reliability  value  will  be 
shown  by  a horizontal  line  corresponding  to  that  value  on  the  vertical 
axis.  The  current  reliability  plots  shall  be  included  in  the  Status 
Report  (See  4.9.1). 

4.7.1  Achieved  reliability.  A plot  of  achieved  reliability  expressed 
as  a point  estimate  shall  be  used  to  depict  the  results  of  the  reliability 
growth  test.  This  plot  shall  be  made  showing  the  cumulative  reliability 
versus  the  cumulative  test  time.  This  plot  shall  not  be  adjusted  by 
negating  past  failures  because  of  present  or  future  design  changes.  This 
curve  shall  be  identified  as  "Achieved  Reliability". 
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4.7.2  Adjusted  reliability.  A second  plot  shall  be  made  to  reflect 
the  level  at  which  the  achieved  reliability  would  be  if  those  failures 
were  discounted  for  which  acceptable  corrective  action  has  resolved  a 
failure  to  the  satisfaction  of  the  procuring  activity.  This  curve  shall 
be  identified  as  "Adjusted  Reliability". 

4.7.3  Moving  average.  A moving  average  of  achieved  reliability  is 
constructed  by  arranging  the  failure  times  (accumulated  test  times  be- 
tween failures)  for  the  eguipment  on  test  in  chronological  order  of 
occurrence.  The  moving  average  for  any  specific  number  of  failures  is 
computed  as  the  arithmetic  mean  of  the  failure  times  selected  sequentially 
and  in  reverse  order.  The  number  of  failures  to  be  used  in  the  computa- 
tion is  arbitrary  but  is  restricted  to  10  or  less.  An  example  is  shown 

in  Appendix  C.  This  curve  shall  be  identified  as  "Moving  Average"  and 
the  number  of  failures  used  for  computation  shall  be  noted. 

4.8  Test  program  reviews.  Test  program  reviews  shall  be  conducted  at 
scheduled  milestones  throughout  the  reliability  development  test  program 
and  shall  be  identified  in  the  approved  test  plan.  Attendance  at  these 
reviews  will  include  procuring  or  user  activity  and  contractor  personnel 
and  their  designated  representatives.  These  reviews  shall  be  documented 
by  the  contractor  and  submitted  as  an  appendix  to  the  Status  Report  (See 
4.9.1). 

4.8.1  Test  readiness  reviews.  A test  readiness  review  shall  be 
planned,  scheduled,  and  conducted  at  least  seven  (7)  days  prior  to  the 
start  of  any  test  to  assure  that  the  test  item  and  all  supporting  elements 
are  ready  to  begin  test.  The  test  readiness  review  shall  include,  to 

the  extent  applicable,  the  following: 

a.  Status  of  design  and  fabrication, 

b.  Results  of  reliability  predictions 

c.  Results  of  all  applicable  previous  tests, 

d.  Summary  of  the  open  problems  and  failures, 

e.  Status  of  problem  and  failure  closures, 

f.  Availability  of  test  procedures  approved  by  appropriate 
personnel  to  indicate  technical  review, 

g.  Status  of  previously  assigned  action  items, 

h.  Assignment  of  action  items  resulting  from  the  review,  and 

i.  Conclusions  of  review  (Test  start  approval/disapproval). 

4.8.2  Status  reviews.  Formal  reviews  shall  be  scheduled  at  preplanned 
milestones  during  the  reliability  development  test  program  to  permit  the 
contractor  and  procuring  or  user  activity  to  review  the  status  of  testing. 
The  status  reviews  shall  be  scheduled  approximately  monthly  and  shall 
include,  to  the  extent  applicable,  the  following: 

a.  Current  reliability  growth  assessments  and  projections, 

b.  Results  of  current  problem  and  failure  investigations  and 
engineering  analyses. 
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c.  Corrective  action  recommendations, 

d.  Potential  design  problems  based  on  the  corrective  action 
recommendations, 

e.  Status  of  subcontractor  and  supplier  reliability  development 
tests, 

f.  Status  of  previously  assigned  action  items,  and 

g.  Assignment  of  action  items  resulting  from  the  review. 

4.8.3  Test  completion  review.  A test  completion  review  shall  be 
scheduled  at  the  completion  of  test.  This  review  shall  be  conducted  to 
evaluate  the  results  of  the  testing  in  compliance  with  the  requirements 
of  the  contract,  equipment  or  model  specification,  and  this  standard. 

This  review  shall  include,  to  the  extent  applicable,  the  following: 

a.  Current  reliability  growth  assessments  and  achievements, 

b.  Status  of  open  problems  and  failures, 

c.  Status  of  corrective  actions, 

d.  Status  of  previously  assigned  action  items, 

e.  Assignment  of  action  items  resulting  from  the  review,  and 

f.  Conclusions  of  the  review. 

4.9  Reports . No  data  shall  be  required  by  this  document  or  the  ref- 
erenced documents  in  paragraph  2 unless  specified  in  the  contract.  Re- 
ports shall  be  in  accordance  with  the  contract. 

4.9.1  Status  report.  The  contractor  shall  prepare  a status  report 
within  seven  (7)  days  from  the  last  date  of  the  scheduled  reporting 
period.  Each  report  shall  present  the  test  results  and  all  pertinent 
data  obtained  durng  the  reporting  period,  the  total  number  of  problems 
and  failures  for  each  operational  mode  specified  in  the  duty  cycle, 
plots  of  measured  and  projected  reliability  growth  assessment  data,  and 
the  estimated  completion  date.  The  minutes  of  the  test  program  reviews 
shall  be  included  as  an  appendix  to  this  report. 

4.9.2  Failure  summary.  The  contractor  shall  prepare  a failure  summary 

report  as  of  the  end  of  each  month.  The  summary  report  shall  be  prepared 
in  three  parts:  Part  I,  cumulative  number  of  failures  to  summary  date; 

Part  II,  failures  for  which  corrective  action  has  not  been  established; 

Part  III,  cumulative  listing  of  closed  reports.  The  contractor's  ex- 
isting failure  summary  may  be  used  with  minimum  changes  necessary  to 
meet  the  requirements  of  this  document. 

4.9.3  Final  report.  A final  report  shall  be  prepared  and  submitted 

by  the  contractor  within  thirty  (30)  days  after  completion  of  all  scheduled 
testing.  This  report  shall  summarize  all  results  obtained  during  the 
reliability  development  test  program,  and  shall  contain  copies  of  all 
test  data  sheets  and  test  logs.  The  report  shall  include,  but  not  be 
limited  to,  a comparison  of  results  with  the  reliability  predictions, 
the  conclusions  drawn,  a summary  of  problems  and  failures,  the  correc- 
tive actions  taken,  and  an  assessment  of  the  reliability  achieved.  The 
final  report  shall  define  the  configuration  of  the  equipment  before  and 
after  incorporation  of  design  changes  resulting  from  the  test  program. 
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The  report  shall  include  a tabulation  of  all  differences  between  the 
requirements  of  the  equipment  specification  and  results  of  the  test 
program,  and  provide  recommendations  for  resolution  of  the  noted  differences. 

Unless  otherwise  specified,  the  final  report  shall  also  contain  maintenance 
times  for  each  failure  or  maintenance  action,  subdivided  into  the  following 
categories,  as  defined  in  MIL-HDBK-472. 


a . 

Local ization. 

b. 

Isolation, 

c. 

Disassembly, 

d. 

Interchange, 

e. 

Reassembly, 

f . 

A1 ignment. 

g. 

Check-out,  and 

h. 

Total 

? 


4.10  Government  furnished  property  (GFP).  Equipment,  facilities,  and 
material  furnished  by  the  government  will  be  specified  in  the  contract. 
Any  GFP  included  in  the  equipment  to  be  tested  will  be  accompanied  by 
applicable  existing  reliability  test  data. 

4.11  Inspection.  Visits  to  the  contractor's  facility  for  assuring 
compliance  with  the  reliability  development  test  program  requirements 
will  be  made  by  procuring  or  user  acitivity  personnel  or  its  designated 
representatives. 
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5.  DETAIL  REQUIREMENTS 

5.1  Test  requirements.  Testing  of  the  systems  and  equipment  shall  be 
as  specified  in  the  equipment  or  model  specification  and  the  contract. 

The  reliability  development  tests  shall  be  planned  in  acco  dar.ce  with 
the  reauirements  of  this  document  to  satisfy  the  development  testing 
reouirements  of  MIL-STD-785  and  to  provide  data  for  the  evaluation  of 
reliability  growth. 

5.1.1  Test  items.  The  test  items  for  the  reliability  development 
tests  shall  be  defined  in  the  Reliability  Development  Test  Plan.  The 
number  of  items  to  be  tested  shall  be  as  specified  in  the  equipment  or 
model  specification. 

5.1.2  Test  length.  The  total  test  time  (cumulative  hours,  cycles, 
rounds,  etc.)  for  the  reliability  growth  tests  shall  be  as  specified  in 
the  equipment  or  model  specification.  The  test  shall  be  a fixed  length 

test  based  on  the  mission  profile  and  the  life  and  operational  characteristics 
of  the  equipment.  Unless  otherwise  specified,  when  two  or  more  test 
items  are  used,  the  minimum  operating  time  for  each  test  item  shall  be 
not  less  than  one-half  the  average  operating  time  for  all  items  on  test. 

5.1.3  Test  cycle  (typical ) . Unless  otherwise  specified,  the  test 
cycle  shown  in  Figure  1 and  described  below,  shall  be  used.  The  parti- 
cular operating  and  storage  temperatures,  vibration  levels,  rates  and 
frequencies  of  change,  and  equipment  ON-OFF  and  duty  cycles  shall  be  as 
specified  in  the  equipment  or  model  specification  and  described  in  the 
test  plan. 

a.  With  test  item  OFF  and  vibration  OFF,  cool  chamber  to  the  low 
storage  temperature  and  allow  equipment  temperature  to  stabilize  (cold 
soak) . 

b.  When  test  item  temperature  has  stabilized  at  the  low  storage 
temperature,  heat  chamber  to  low  operating  temperature,  turn  test  item 
ON  and  perform  specified  number  of  preoperational  checks  or  run-ups. 

c.  Upon  completing  the  preoperational  checks,  heat  chamber  to  high 
operating  temperature  with  test  item  ON  and  vibration  ON  and  commence 
specified  duty  cycle  for  time  specified,  or  if  no  time  is  specified,  for 
two  (2)  hours.  Unless  otherwise  specified,  the  chamber  temperature 
shall  be  cycled  between  the  upper  and  lower  operating  temperatures  of 
the  equipment  during  this  portion  of  the  test  cycle  at  the  specified 
rate  and  frequency  of  change. 

d.  Upon  completing  the  duty  cycle  period,  turn  test  item  and 
vibration  OFF,  heat  chamber  to  the  high  storage  temperature,  and  allow 
test  item  to  stabilize  (hot  soak). 

e.  When  test  item  temperature  has  stabilized,  cool  chamber  to  high 
operating  temperature,  turn  item  ON  and  perform  the  specified  number  of 
preoperational  checks  or  run-ups. 
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f.  Upon  completing  specified  number  of  preoperational  checks,  cool 
chamber  to  the  low  operating  temperature  with  test  item  ON  and  vibration 
ON  and  commence  specified  duty  cycle  for  the  time  specified,  or  if  no 
time  is  specified,  for  two  (2)  hours.  Unless  otherwise  specified,  the 
chamber  temperature  shall  be  cycled  between  the  upper  and  lower  operating 
temperatures  during  this  portion  of  the  test  cycle  at  the  specified  rate 
and  frequency  of  change. 

g.  Upon  completing  the  duty  cycle  period,  turn  test  item  and 
vibration  OFF  and  repeat  cycle  from  (a)  above. 

5.2  Other  specified  tests.  The  data  acquired  during  other  tests 
described  in  the  equipment  or  model  specification  shall  be  used  to 
monitor  reliability  growth.  The  data  obtained  from  all  testing,  in- 
cluding component  and  weapons  replaceable  assembly  (WRA)  testing,  shall 
be  included  in  the  planned  failure  reporting  and  corrective  action 
system.  Failure  modes  associated  with  system  interfaces  that  are  identified 
during  testing  shall  be  reported. 


Preparing  Activity: 
Navy  - AS 

(Project  MISC  NB59) 
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flGURE  1 - TEST  CYCLE  (TYPICAL) 
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NOTES 


Supersession  Note. 

The  following  Aeronautical  Requirements  documents  are  superseded  by 


MIL -STD-  2068 

(AS)  dated  21  March  1977 

No. 

Title 

Date 

AR-104 

Reliability  Development  Test  for  Avionic 
Equipment 

30  April  1974 

AR-108 

Fluid  and  Mechanical  Airframe  Subsystems 
and  Airborne  Special  Mission  Systems;  Re- 
liability Development  Test  Program  for 

8 

November 

1974 

AR-111 

Reliability  Development  Test  Program 
for  Armament  Equipment,  Gun  Systems, 
and  Associated  Stores  Management  Systems 

8 

November 

1974 

AR-112 

Reliability  Development  Test  Program 
for  Crew  Systems 

0 

November 

1974 

AR- 113 

Reliability  Development  Test  Program 
for  Ground  Support  Equipment 

8 

November 

1974 

AR- 114 

Reliability  Development  Test  Program 
for  Range  Instrumentation  Equipment 

15 

November 

1974 

AR-115 

Reliability  Development  Test  Program 
for  Ship  Installation  Equipment 

15 

November 

1974 

AR -11 6 

Reliability  Development  Test  Program 
for  Photographic  Equipment 

15 

November 

1974 

AR- 1 1 7 

Reliability  Development  Test  Program 
for  Meteorological  Equipment 

15 

November 

1974 

AR - 1 18 

Reliability  Development  Test  Program 
for  Missile  Equipment 

22 

February  1976 
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APPENDIX  A 

RELIABILITY  PROGRAM  TASKS 
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21  March  1977 


10.  GENERAL 

10.1  Scope.  This  appendix  describes  reliability  program  tasks  that 
shall  be  completed  and  approval  obtained  from  the  procuring  activity 
prior  to  the  start  of  the  reliability  development  test  program. 

10.2  Appl ication.  Each  task  described  under  50.1  is  presented  as  a 
guide  only.  It  is  incumbent  on  all  concerned  to  specify  a reliability 
program  in  accordance  with  MIL-STD-785  and  to  require  a reliability  pro- 
gram plan  for  approval. 

20.  REFERENCED  DOCUMENTS 


20.1  Issues  of  documents.  The  following  documents  of  the  issue  in 
effect  on  date  of  invitation  for  bids  0 trequest  for  proposal,  form  a 
part  of  this  appendix  to  the  extent  specified  therein. 


STANDARDS 

MILITARY 

MI L -STD-7 56  Reliability  Prediction 

HANDBOOKS 

MILITARY 


MIL-HDBK-217  Reliability  Prediction  of  Electronic  Equipment 


PUBLICATIONS 

AD-821640 


Rome  Air  Development  Center  (RADC)  Reliability  Notebook 


RADC-TR-73-248 


Dormancy  and  Power  On-Off  Cycling  Effects  on  Electronic 
Equipment 


RADC-TR-75-22  Non-El ectronic  Reliability  Notebook 


(Copies  of  specifications,  standards,  drawings,  and  publications 
required  by  contractors  in  connection  with  specific  procurement  functions 
should  be  obtained  from  the  procuring  activity  or  as  directed  by  the 
contracting  officer.) 


30.  DEFINITIONS 

30.1  Derating  criteria.  A standard  that  established  the  ratio  of 
maximum  applied  stress  to  rated  stress  for  parts  applications. 

40.  GENERAL  REOUIREMENTS 

40.1  Reliability  program  tasks. 
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40.1.1  Mission  profile.  This  task  is  accomplished  early  in  the  con- 
ceptual phase  and  is  used  to  define  the  operational  modes  of  the  system. 

40.1.2  Environmental  profile.  This  task  follows  the  mission  profile 
definition  and  is  used  to  define  the  worst  case  environments  during  the 
life  of  a system. 

40.1.3  Reliability  block  diagram.  This  task  is  accomplished  during 
initial  design  after  the  system  coi  figuration  is  defined. 

40.1.4  Rel iabi 1 i ty  mathemat ■ cal  model.  This  task  follows  the  reli- 
ability block  diagram  and  is  prepared  by  a review  of  the  block  diagram. 
Reliability  predictions  are  used  as  inputs  for  solution  of  the  mathe- 
matical model. 

40.1.5  Reliability  predictions.  This  task  is  accomplished  during 
design  and  the  results  are  used  to  guide  the  design. 

40.1.6  Reliability  allocations.  This  task  is  accomplished  with  the 
reliability  prediction  task  and  is  used  to  allocate  system  failure 
allowances  among  the  system  elements. 

40.1.7  Fai 1 ure  mode,  effects , and  criticality  analysis  (FMECA) . 

This  task  is  accomplished  early  in  the  design  phase  and  is  used  to 
identify  potential  failures  inherent  in  the  equipment. 

40.1.8  Stress  analysis.  This  task  is  accomplished  during  the  detail 
design  and  is  used  to  identify  maximum  actual  stresses  and  to  guide  the 
design. 

40.1.9  Morse  case  analysis.  This  task  is  accomplished  during  the 
detail  design  and  is  used  to  identify  any  detrimental  output  responses 
when  parts  or  components  are  considered  to  be  at  detrimental  tolerance 
1 imits. 

40.1.10  Sneak  circuit  analysis.  This  task  is  accomplished  during 
detail  design  to  identify  latent  paths  in  the  design  that  can  result  in 
undersirable  responses. 

50.  DETAIL  REQUIREMENTS 

50.1  Development  of  reliability  program  tasks.  The  reliability 
program  tasks  must  be  developed  early  in  the  conceptual  and  development 
phases  of  a program  and  shall  be  approved  by  the  procuring  activity 
prior  to  the  start  of  the  reliability  development  tests. 

50.1.1  Mission  profile.  A graphic  or  tabular  presentation  describing 
equipment  performance  requirements  as  a function  of  time  and  showing  the 
occurrences  of  scheduled  events,  their  sequence  and  duration,  during  a 
complete  operational  mission.  The  mission  profile  is  used  to  define  the 
equipment  duty  cycle  and  to  determine  realistic  operational  test  levels 
to  be  applied  during  the  reliability  development  test. 
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50.1.2  Environmental  profile.  A graphic  or  tabular  presentation  of 
the  worst  case  environments  (both  natural  and  induced)  that  an  equipment 
will  be  subjected  to  from  acceptance  to  the  end  of  useful  life.  This 
profile  is  used  to  determine  the  realistic  environmental  test  levels  to 
be  applied  during  the  reliability  development  test. 

50.1.3  Reliability  block  diagram.  A graphic  analog  of  logical  events 
that  result  in  the  success  or  failure  of  the  hardware.  The  block  diagram 
is  the  first  step  in  modeling  system  reliability. 

50.1.4  Reliability  mathematical  model.  An  algebraic  analog  of  the 
block  diagram  that  is  prepared  by  reviewing  the  block  diagram.  The 
mathematical  model  is  used  to  determine  the  reliability  of  the  system 
and  items  hereof. 

50.1.5  Reliability  predictions.  An  analytical  estimation  of  the 
numerical  reliability  of  an  equipment  postulated  on  the  analysis  of 
realistic  and  conservative  failure  rate  data.  This  prediction  shall  be 
done  in  accordance  with  the  design  prediction  procedure  of  MIL-STD-756 
for  each  distinct  mission  phase  or  mode  of  operation.  The  prediction 
shall  be  based  on  failure  rate  data  from  sources  such  as  MIL-HDBK-21 7 , 

RADC  Reliability  notebook,  RADC-TR-73-248,  and  RADC-TR-75-22.  Other 
realistic  and  conservative  failure  rate  data  sources,  including  con- 
tractor in-house  data,  can  be  used  but  requires  prior  approval  of  the 
procuring  activity.  The  prediction  report  shall  be  approved  by  the 
procuring  activity  prior  to  the  start  of  testing. 

50.1.6  Reliability  allocation.  The  numerical  reliability  requirement 
for  the  equipment  shall  be  subdivided  and  allocated  to  each  item  within 
the  equipment.  The  rationale  supporting  the  allocations  shall  be  approved 
by  the  procuring  activity. 

50.1.7  Failure  mode,  effects,  and  criticality  analysis  (FMECA).  A 
systematic  procedure  for  examining  and  classifying  all  the  ways  in  which 
a system  may  fail,  together  with  the  effects  of  the  failure  modes  on  the 
system  and  their  consequences  to  performance  of  the  mission.  The  FMECA 
should  be  carried  down  to  the  component  and  circuit  level  where  the 
effects  of  individual  part  failures  can  be  studied. 

50.1.8  Stress  analysis.  An  analysis  conducted  to  identify  the  maxi- 
mum actual  stresses  (e. g.  , thermal,  electrical,  mechanical)  induced  on  a 
part  in  its  application.  The  results  of  this  analysis  are  compared  with 
the  parts  derating  criteria  to  determine  overstress  conditions. 

50.1.9  Worst  case  analysis.  An  analysis  conducted  to  determine  that 
a design  will  give  proper  response  for  all  allowable  inputs  with  all 
components  considered  to  be  at  the  most  detrimental  initial  or  end  of 
life  tolerance  limits  for  all  specified  environmental  conditions. 

50.1.10  Sneak  circuit  analysis.  An  analysis  conducted  to  determine 
any  latent  path  in  a design  which  may  result  in  an  undersirable  response 
to  an  allowed  input  under  planned  operating  conditions. 
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50.2  Other  reliability  design  techniques.  The  use  of  redundance, 
stress-strength  analysis,  and  other  reliability  oriented  design  techniques 
shall  be  used  in  designing  equipment  to  enhance  equipment  reliability. 


APPENDIX  B 

FAILURE  CLASSIFICATION  FOR 
RELIABILITY  GROWTH  ASSESSMENT 
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10.  GENERAL 

10. 1 Scope.  This  appendix  establishes  criteria  for  classification  of 
failure  occurring  during  the  reliability  development  tests. 

10.2  Appl ication.  The  ground  rules  and  scoring  criteria  included  in 
this  appendix  shall  be  used  to  classify  failures  as  either  relevant  or 
nonrelevant  for  the  purpose  of  assessing  reliability  growth  of  the 
equi pment. 

20.  REFERENCED  DOCUMENTS 
Not  Applicable. 

30.  DEFINITIONS 

30.1  Failures.  In  addition  to  the  definition  of  failure  as  given  in 
MIL-STD-721,  and  any  definition  given  in  the  applicable  reliability  test 
specification  and  this  standard,  the  following  criteria  for  the  determina- 
tion of  a failure  shall  apply: 

Whenever  any  of  the  performance  characteristics  are  outside  of  the 
requirements  of  the  equipment  or  model  specification  at  any  specified 
environmental  conditions,  at  least  one  failure  has  occurred. 

40.  GENERAL  REQUIREMENTS 

40.1  Failure  reporting.  All  problems  and  failures  and  the  conditions 
under  which  they  occurred  shall  be  reported  and  investigated  in  accordance 
with  the  requirements  of  this  standard. 

40.2  Failure  classification.  All  failures  occurring  during  reliability 
development  tests,  including  failures  occurring  during  equipment  burn-in 
under  the  environmental  conditions  specified  for  the  reliability  develop- 
ment test,  shall  be  classified  and  reported  as  either  relevant  or  non- 
relevant. Only  those  failures  occurring  during  the  reliability  develop- 
ment tests  and  classified  relevant  shall  be  used  in  computing  equipment 
mean  time  between  failure  (MTBF)  and  for  making  reliability  growth 
assessments . 


40.3  Relevant  failures.  All  failures  are  relevant  unless  determined 
by  the  procuring  or  user  activity,  or  an  authorized  representative 
thereof,  to  be  caused  by  a condition  external  to  the  equipment  under 
test  which  is  not  a test  requirement. 


40.3.1  Design  failures.  Failures  due  to  design  deficiencies  of 
either  the  equipment  or  component  parts  shall  be  classified  relevant. 
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40.3.2  Workmanship  failures.  Failures  resulting  from  workmanship  or 
quality  control  shall  be  classified  as  relevant. 

40.3.3  Component  part  failures.  Failures  due  to  defective  component 
parts  shall  be  classified  as  relevant  failures.  Each  independent  failure 
of  component  parts  of  the  same  type  shall  be  considered  a separate 
relevant  failure. 

40.3.4  Wearout  parts.  Certain  parts  of  known  limited  life,  such  as 
batteries,  may  have  a life  stipulated  prior  to  the  initiation  of  testing 
as  approved  by  the  procuring  activity.  Failures  of  these  parts  occurring 
prior  to  the  end  of  the  stipulated  period  are  relevant.  Failures  of 
these  parts  occurring  after  the  stipulated  period  are  nonrelevant,  but 
any  dependent  failures  caused  thereby  are  relevant. 

40.3.5  Multiple  failures.  In  the  event  simultaneous  part  failures 
occur,  each  failed  part  which  would  independently  prevent  satisfactory 
equipment  performance  shall  be  counted  as  a relevant  failure. 

40.3.6  Intermittent  operation.  The  second  and  subsequent  occurrences 
of  an  intermittent  operation  on  any  one  item  under  test  shall  be  counted 
as  a relevant  failure.  (See  40.4.5). 

40.3.7  Adjustments . Each  adjustment  of  a control  which  could  not 
readily  be  accomplished  during  a mission  shall  be  classified  as  a relevant 
fail ure. 

40.3.8  Built-in  test  failures.  A failure  of  the  built-in  test  (BIT) 
functions  that  degrades  sytem  performance  beyond  the  acceptable  limits 
shall  be  classified  relevant.  False  alarms  of  BIT  functions  shall  be 
classified  as  relevant  failures. 

40.3.9  Other  failures.  All  other  failures  not  specifically  defined 
as  nonrelevant  shall  be  classified  as  relevant. 

40.4  Nonrelevant  failures.  Although  nonrelevant  failures  are  not 
used  for  MTBF  calculations  or  for  reliability  growth  assessments,  all 
failures  shall  be  recorded  and  investigated. 

40.4.1  Improper  test  installation.  Failures  directly  attributable  to 
improper  installation  in  the  test  facility  shall  be  classified  nonrelevant. 

40.4.2  Test  instrumentation  failures.  Failures  directly  attributable 
to  a failure  of  the  test  instrumentation  or  monitoring  equipment,  except 
BIT,  shall  be  classified  nonrelevant. 

40.4.3  Test  operator  errors.  Failures  resulting  from  test  operator 
error  in  setting  up  or  in  testing  the  equipment  shall  be  classified 
nonrel evant. 

40.4.4  Test  procedure  errors.  Failures  resulting  from  an  error  in 
the  test  procedures  shall  be  classified  nonrelevant. 
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40.4.5  Intermittent  operation.  The  first  occurrence  of  an  intermittent 
operation  on  any  one  item  under  test  shall  be  classified  as  nonrelevant. 

(See  40.3.6). 

40.4.6  Repair  failures.  Failures  that  result  from  trouble-shooting, 
repair  verification  or  scheduled  maintenance  shall  be  classified  nonrelevant 
if  the  failure  occurs  during  repair  and  verification  as  described  in  the 
test  plan. 

50.  DETAIL  REQUIREMENTS 
Not  ap pi icable. 
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APPENDIX  C 

EXPLANATORY  INFORMATION 


10.  GENERAL 

10.1  Scope.  This  appendix  contains  explanatory  information. 

10.2  Appl ication.  The  information  in  this  appendix  is  for  the  guidance 
of  the  procuring  activity  and  does  not  contain  direct  requirements  of 
this  standard. 


20.  REFERENCED  DOCUMENTS 
Not  appl icabl e. 

30.  DEFINITIONS 

Not  applicable. 

40.  GENERAL 


40.1  Data  requirements.  Data  items  generated  in  accordance  with  this 
standard  are  not  deliverable  unless  specified  on  the  Contract  Data 
Requirements  List  (DD  Form  1423)  or  the  contract  schedule.  This  standard 
applies  to,  but  is  not  restricted  to,  the  following  Data  Item  Descriptions 
(DD  Form  1664). 


a.  UDI-T-22723 

b.  UDI-R-21 1 34 

c.  UDI-R-21140 

d.  UDI-R-21 1 35 

e.  UDI-R-21 1 39 

f.  UDI-R-21 1 36 


Thermal  Survey  Report 

Report,  Reliability  and  Maintainability  Prediction 

Report,  Analysis,  Failure  Modes  and  Effects 

Plan,  Reliability  Test 

Report,  Failure  Summary 

Report,  Reliability  Test  Results 


40.2  Moving  average  example.  Table  I is  an  example  of  the  computation 
of  moving  average  utilizing  1,  2,  3,  ...  6 failures  in  the  computation. 

The  moving  average  is  constructed  by  arranging  the  failure  times  (accumulated 
test  time  between  failures)  for  the  items  on  test  in  chronological  order 
of  occurrence.  The  moving  average  for  any  given  number  of  failures  is 
computed  as  the  arithmetic  mean  of  that  number  of  failure  times  selected 
sequentially  and  in  reverse  order.  For  example,  the  moving  average  of 
two  failures  is  obtained  by  adding  the  last  two  failure  times  and  dividing 
by  two;  for  three  failures,  by  summing  the  last  three  failure  times  and 
dividing  by  three;  etc.  The  number  of  failures  used  in  the  computation 
is  arbitrary  but  should  be  restricted  to  ten  or  less.  Initially,  some 
moving  averages  cannot  be  computed  since  enough  failures  have  not  occurred. 
For  example,  the  moving  average  utilizing  the  last  six  times  between 
failures  cannot  be  computed  until  six  failures  have  occurred.  The 
failure  times  were  selected  arbitrarily  for  this  example.  The  times 
between  failures  are  arranged  in  chronological  order  and  indexed  by  the 
cumulative  failure  count.  The  cumulative  test  time  is  the  total  of  all 
time  between  failures  up  to  and  including  the  failure  time  indexed. 
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TABLE  I 

MOVING  AVERAGE 


Cum.  Cum.  Time  Cum. 
Fail.  Test.  Between  MTBF 
Count  Time  Fail. 


Moving  MTBF 

Using  Last  (N)  Failures 
(N  is  Table  Index) 


1 2 3 4 5 6 


1 

1 

1 

1 

1 

0 

0 

0 

0 

0 

2 

4 

3 

2 

3 

2 

0 

0 

0 

0 

3 

8 

4 

3 

4 

4 

3 

0 

0 

0 

4 

13 

5 

3 

5 

5 

4 

3 

0 

0 

5 

20 

7 

4 

7 

6 

5 

5 

4 

0 

6 

30 

10 

5 

10 

9 

7 

7 

6 

5 

7 

42 

12 

6 

12 

11 

10 

9 

8 

7 

8 

57 

15 

7 

15 

14 

12 

11 

10 

9 

9 

78 

21 

9 

21 

18 

16 

15 

13 

12 

10 

104 

26 

10 

26 

24 

21 

19 

17 

15 

11 

136 

32 

12 

32 

29 

26 

24 

21 

19 

12 

177 

41 

15 

41 

37 

33 

30 

27 

25 

13 

228 

51 

18 

51 

46 

41 

38 

34 

31 

14 

292 

64 

21 

64 

58 

52 

47 

43 

39 

15 

372 

80 

25 

80 

72 

65 

59 

54 

49 

16 

473 

101 

30 

101 

91 

82 

74 

67 

62 

17 

599 

126 

35 

126 

114 

102 

93 

84 

77 

18 

757 

158 

42 

158 

142 

128 

116 

106 

97 

19 

956 

199 

50 

199 

179 

161 

146 

133 

121 

20 

1205 

249 

60 

249 

224 

202 

183 

167 

152 

21 

1518 

313 

72 

313 

281 

254 

230 

209 

191 

22 

1879 

361 

85 

361 

337 

308 

281 

256 

234 

23 

2262 

383 

98 

383 

372 

352 

327 

301 

277 

24 

2668 

406 

111 

406 

395 

383 

366 

342 

319 

25 

3099 

431 

124 

431 

419 

407 

395 

379 

357 

40.3  Test  time.  One  of  the  main  purposes  of  the  reliability  development 
test  is  to  design  out  failure  causes.  This  process  will  necessarily  involve 
time  to  detect  and  analyze  failures  as  well  as  redesign  and,  for  test  pur- 
poses, fabricate  hardware.  Thus,  for  slow  occurrences  of  failures  in  test 
the  critical  variable  is  test  time  while,  if  the  redesign  and  resulting 
hardware  change  takes  considerable  time,  then  calendar  time  becomes  the 
significant  variable.  The  test  duration  must  be  specified  in  the  equipment 
or  model  specification  and  should  be  based  on  an  analysis  of  the  mission 
profile,  operational  and  life  characteristic  of  the  item,  and  the  program 
requirements,  (e.g.,  budget,  schedule,  resources,  etc.).  The  reliability 
development  test  should  be  planned  as  a fixed  length  test  and  the  test 
duration  must  be  specified.  Fixed  length  test  times  of  10  to  25  multiples 
of  the  specified  MTBF  will  generally  provide  a test  length  sufficient  to 
achieve  sufficient  to  achieve  the  desired  reliability  growth  for  equip- 
ment in  the  50  to  2000  hours  range.  For  equipments  over  2000  hours,  test 
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lengths  should  be  based  on  equipment  complexity  and  the  needs  of  the  pro- 
gram, but  as  a minimum,  should  be  one  multiple  of  the  specified  MTBF. 

In  any  event,  the  test  length  should  not  be  less  than  2000  hours  or  more 
than  10,000  hours. 

40.4  Corrective  action.  Corrective  action  shall  have  been  developed 
for  all  failures  experienced  in  the  Reliability  Development  Tests  and 
such  corrective  action  shall  have  been  incorporated  in  the  equipment  to 
be  subjected  to  the  Reliability  Demonstration  required  by  MIL-STD-785. 

In  aircraft  and  missile  programs,  no  flight  testing  shall  be  performed 
until  after  the  Reliability  Development  Tests  have  been  completed  and 
no  aircraft  or  missiles  shall  be  delivered  to  operating  units  until 
corrective  action  for  all  failures  experienced  during  the  Reliability 
Development  Tests  have  been  incorporated. 

40.5  Specification  requirements.  Equipment  or  model  specifications  shall 
contain  the  following: 

a.  Test  levels  (see  4.3). 

b.  Temperature  (see  4.3.1). 

c.  Vibration  (see  4.3.2). 

d.  On-off  cycling  (see  4.3.3.1). 

e.  Duty  cycle  (see  4. 3. 3. 2). 

f.  Performance  parameters  (see  4.5). 

g.  Performance  measurements  (see  4.5.3). 

h.  Test  requirements  (see  5.1). 

j.  Test  length  (see  5.1.2). 

k.  Test  cycle  (see  5.1.3). 

50.  DETAIL  REQUIREMENTS 
Not  applicable 
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CALIFORNIA  STATE  UNIVERSITY,  NORTHRIDGE 

research  in  testing  and  evaluation  methodology 


SCHOOL  OF  ENGINEERING 
AND  COMPUTER  SCIENCE 
(21)1  88S-218J 


CALIFORNIA  STATE  UNIVERSITY,  NORTHRIDGE 
Northridge,  California  91324 
"IN  THE  SAN  FERNANDO  VALLEY" 


RESEARCH  IN  TESTING  AND  EVALUATION  METHODOLOGY 


Testing  and  Evaluation  (T&E)  is  an  engineering  activity  of 
major  economic  importance  to  which  little  or  no  attention  has  been  given 
in  engineering  academic  programs  or  research.  A number  of  corporations 
and  government  agencies  have  carried  on  extensive  T&E  functions  for  some 
time.  Recent  changes  n laws  concerning  government  purchasing  and  con- 
sumer product  liability  will  undoubtedly  extend  the  interest  in  such  ac- 
tivities. Accordingly,  there  will  be  an  increasing  demand  for  attention 
to  T&E  by  the  engineering  profession  and  their  academic  community. 

During  the  past  four  years,  the  School  of  Engineering  and 
Computer  Science  at  California  State  University,  Northridge,  has  under- 
taken several  activities  associated  with  T&E.  The  School  has  been  in- 
volved with  the  U.  S.  Navy  Pacific  Missile  Test  Center  at  Pt.  Mugu, 
California  in  developing  a graduate  engineering  program  with  emphasis  in 
Testing  and  Evaluation.  Although  this  program  leans  heavily  toward  the 
soecific  interests  of  the  U.  S.  Navy  and  much  of  the  background  material 
is  derived  from  military  and  other  government  agency  experience  with  T&E, 
there  appears  to  be  significant  interest  among  engineers  working  within 
the  civilian  economic  sector  center  for  a similar  program  if  it  were  not 
so  heavily  oriented  toward  the  military.  Furthermore,  if  T&E  is  such  a 
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significant  engineering  activity  as  preliminary  analysis  would  indicate, 
some  attention  should  be  given  to  its  normal  inclusion  in  engineering  cur- 
ricula. For  example,  Testing  and  Evaluation  would  appear  to  be  an  integral 
part  of  the  design  process,  although  it  is  invariably  neglected  in  teaching 
design.  However,  to  pursue  further  the  development  of  T&E  within  engineer- 
ing academia,  a thorough  study  of  Testing  and  Evaluation  as  practiced 
throughout  industry,  especially  in  the  civilian  sector,  must  be  undertaken. 

During  the  1975-1976  academic  year,  a research  program  on  the 
California  State  University,  Northridge  campus  did  investigate  and  analyze 
T&E  practices  within  the  U.  S.  Navy,  resulting  in  a series  of  monographs 
ordering  portions  of  Naval  philosophy  and  practice.  As  part  of  that  study, 
major  U.  S.  corporations  were  surveyed  to  assess  their  interest  and  in- 
volvement with  T&E.  A copy  of  those  survey  results  are  attached.  The  re- 
sponses did  indicate  a keen  interest  and  extensive  involvement,  though 
practices  probably  varied  widely  between  different  organizations.  Undoubt- 
edly there  is  an  extensive  methodology  of  T&E  employed  throughout  industry 
and  government,  but  it  does  not  appear  to  be  collected  and  available  through 
any  single  source.  This  is  particularly  evident  if  one  is  concerned  with 
conceptual  advances  and  managerial  philosophy  in  Testing  and  Evaluation  as 
opposed  to  specific  results  of  a test  of  a particular  piece  of  hardware. 

The  School  of  Engineering  and  Computer  Science  has  also  sponsored 
two  seminars  in  an  effort  to  encourage  formal  dialogues  in  Testing  and  Evalu- 
ation. The  first  of  these  seminars  initiated  an  effort  to  perceive  the  T&E 
needs  of  the  public  and  private  sectors  and  to  provide  representatives  of 
those  sectors  with  information  regarding  the  role  that  academic  institutions 
might  play  in  the  development  of  T&E  as  a recognized  discipline.  Papers  au- 
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thored  by  members  of  the  T&E  community  were  presented  at  the  second  seminar. 
Additionally,  faculty  at  the  School  have  instituted  informal  dialogues  v/ith 
several  members  of  the  T&E  community. 

At  this  point,  we  would  propose  mounting  a research  effort  in  the 
methodology  of  Testing  and  Evaluation.  Such  a study  should  include  at  least: 

1.  Extensive  search  for  all  literature  pertinent  to  T&E 
and  the  compilation  of  a T&E  data  bank  or  bibliography. 

2.  An  extensive  examination  of  T&E  methodologies  through- 
out industry  and  government,  including  on-site  visits 
to  a variety  of  organizations  and  industries  to  deter- 
mine the  what,  how,  and  why  of  their  T&E  programs.' 

3.  A functional  analysis  of  the  collected  information  to 
define  any  definite  patterns  or  commonality;  i.e.,  to 
organize  the  material  into  a consistent  body  of  know- 
ledge. 

4.  An  analysis  of  the  information  to  determine  what  spe- 
cial areas  of  knowledge  or  skills  are  needed  in  T&E 
practice,  and  which  of  those  should  be  incorporated 
into  engineering  curriculum. 

5.  A recommendation  for  Testing  and  Evaluation  curricula 
as  part  of  undergraduate  and  graduate  engineering 
programs . 


We  think  such  a project  could  be  completed  successfully  and  that 
the  results  would  be  beneficial  to  the  engineering  profession,  education, 
and  industry.  Within  the  School  of  Engineering  and  Computer  Science  at 
California  State  University,  Northridge,  we  have  more  experience  in  this 
field  than  will  be  found  at  any  other  academic  institution.  Furthermore, 
from  our  experience  in  T&E,  we  have  developed  contacts  with  many  of  the 
current  practitioners  who  are  interested  and  willing  to  help.  All  that 
remains  is  to  obtain  adequate  funding  to  undertake  such  an  extensive  effort. 


January,  1977 


CALIFORNIA  STATE  UNIVERSITY,  NORTHRIDGE 

Northridge,  California  91324 
"IN  THE  SAN  FERNANDO  VALLEY" 


SCHOOL  OF  ENGINEERING 
AND  COMPUTER  SCIENCE 
(213)  885  2183 


June,  1976 


Dear  Sir: 


We  wish  to  share  with  you  some  preliminary  results  of  a survey  of  U.S.  industry 
regarding  the  field  of  Testing  and  Evaluation  that  we  conducted  in  the  latter 
part  of  1975  and  continued  into  the  early  part  of  this  year.  We  find  these 
preliminary  results  most  illuminating.  The  field  of  Testing  and  Evaluation  is 
definitely  becoming  increasingly  important  in  major  U.S.  firms. 

As  a result  of  the  survey  responses,  we  held  a one-day  seminar  for  local  industry 
executives  in  January  and  were  able  to  further  determine  the  needs  of  the  Testing 
and  Evaluation  community.  These  and  other  research  activities  have  evolved  into 
a short  course  on  Testing  and  Evaluation  to  be  offered  in  July  of  this  year. 

A brochure  is  enclosed  and  we  hope  that  you  will  join  us. 


WHO  DOES  THE  TESTING  ? 

Variety  of  Organizations 

Own  Company  Only 

Vendors 

Customers 

Testing  Agencies 

WHAT  IS  TESTED  ? 

Final  Products  and  Parts 
Final  Products  and  Services 
Other  Elements 
Final  Products  Only 
Parts  Only 
Services  Only 

WHERE  IS  TESTING  DONE  ? 
Engineering 
Quality  Control 
Manufacturi ng 
Marketing 
Other 

WHY  IS  TESTING  DONE  ?~ 

Combination  of  Quality,  Cost, 
Time,  and  Quantity 

Quality  Only 

Cost  Only 

Time  Only 

Quantity  Only 

WHEN  IS  TESTING  DONE  ? 

During  Design  and  Development 
During  Production 
After  Production 
In  Decision  Stage 

WHAT  IS  THE  STATURE  OF  T & E 
IN  YOUR  FIRM  ? 

Rising 

Constant 

Variable 

Decl ining 


Percent  of  Respondents 
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WHO  USES  THE  RESULTS  OF  T £ E ? 
Engineering 
Quality  Control 
Top  Management 
Manufacturing 
Other 

WHAT  CRITERIA  ARE  USED  FOR  T & E ? 
Variety  of  Elements 
Internal  & External  Specifications 
Internal  Specifications 
Customer  Needs 
Standard  Procedures 
External  Specifications 
Other  Means 

HOW  IS  TEST  DATA  RECORDED  ? 

Written  Record 
By  Instruments 
By  Computer 
By  Other  Means 

WHO  VALIDATES/AUDITS  THE  TEST  DATAT" 
Test  Engineers 
Management 
Design  Engineers 
Test  Technicians 
PI anners  & Others 

HOW  IS  TESTING  EVALUATED  ? 

Performance  Criteria 
Design  Criteria 
By  Standards 
Statistical  Methods 
Co/ No  Go  Decisions 
Subjective  Processes 
Other  Methods 

HOW  ARE  RESULTS  REPORTED  ? 

Written  Form 
Computer  Form 
Verbal ly 
Other 
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PERCEPTION  OF  BENEFITS  OF  T & E 
Quality  Maker 
Customer  Satisfier 
Confidence  Maker 
Cost  Saver 
Time  Saver 
Quantity  Saver 
Necessary  Evil 
Boss  Pleaser 
Other  Benefits 


CRITICISMS  OF  T & E 


Takes  Too  Long 
Costs  Too  Much 
Inadequate  Methods 
Quantity  of  Personnel 
Errors 
Poor  Data 

Quality  of  Personnel 
Inadequate  Equipment 
Other  Drawbacks 


100 


* Based  on  points  allocated  by  importance  ranks  which  have  been 
normalized  by  the  maximum  score 


GOVERNMENT 


INDUSTRY 


FOR  FURTHER  INFORMATION  REGARDING  THIS  AND 
OTHER  RESEARCH  IN  THE  FIELD  OF  TESTING  AND 
EVALUATION,  CONTACT 

Charles  F.  Sanders,  Dean 
School  of  Engineering  and  Computer  Science 
California  State  University,  Northridge 
Northridge,  California  91324 


